ROMIAE (Rule-Out Acute Myocardial Infarction Using Artificial Intelligence Electrocardiogram Analysis) trial study protocol: a prospective multicenter observational study for validation of a deep learning–based 12-lead electrocardiogram analysis model for detecting acute myocardial infarction in patients visiting the emergency department
Article information
Abstract
Objective
Based on the development of artificial intelligence (AI), an emerging number of methods have achieved outstanding performances in the diagnosis of acute myocardial infarction (AMI) using an electrocardiogram (ECG). However, AI-ECG analysis using a multicenter prospective design for detecting AMI has yet to be conducted. This prospective multicenter observational study aims to validate an AI-ECG model for detecting AMI in patients visiting the emergency department.
Methods
Approximately 9,000 adult patients with chest pain and/or equivalent symptoms of AMI will be enrolled in 18 emergency medical centers in Korea. The AI-ECG analysis algorithm we developed and validated will be used in this study. The primary endpoint is the diagnosis of AMI on the day of visiting the emergency center, and the secondary endpoint is a 30-day major adverse cardiac event. From March 2022, patient registration has begun at centers approved by the institutional review board.
Discussion
This is the first prospective study designed to identify the efficacy of an AI-based 12-lead ECG analysis algorithm for diagnosing AMI in emergency departments across multiple centers. This study may provide insights into the utility of deep learning in detecting AMI on electrocardiograms in emergency departments.
Trial registration
ClinicalTrials.gov identifier: NCT05435391. Registered on June 28, 2022.
INTRODUCTION
Cardiovascular diseases such as acute myocardial infarction (AMI) and acute heart attacks are the leading causes of death worldwide, and it has been confirmed that 31.8% of deaths globally are due to cardiovascular disease. As of 2019, 18.6 million people worldwide have died from cardiovascular diseases, and by 2030 approximately 23.5 million people will die [1,2]. AMI has been reported to be the main cause of cardiovascular disease. Early AMI diagnosis is critical for reducing the incidence of complications and mortality by allowing a rapid reperfusion strategy, thereby reducing medical costs [3]. According to the 2020 international guidelines, the first important step in the initial treatment of patients with suspected AMI is to acquire a standard 12-lead electrocardiogram (ECG) test within 10 minutes after arriving at the emergency medical center and, if possible, immediately contact a specialist [4]. However, with non–ST-segment elevation (NSTE) acute coronary syndrome (ACS), ECG readings may be normal in approximately 30% of patients, even when immediately analyzed by an expert [4]. Expert reading is strongly recommended; however, depending on the region and medical environment, even if an ECG is acquired within 10 minutes of arrival, a cardiologist or emergency medicine specialist may be unavailable. Patient overcrowding is common even in high-level emergency medical centers. Additionally, owing to the scarcity of available human resources during the COVID-19 epidemic, the readability of ECG tests may be poor [5,6].
With the accelerating transformation of digital health, it is critical to utilize artificial intelligence (AI) to quickly diagnose a myocardial infarction. Accordingly, AI research is being introduced for fast and accurate ECG analysis of patients with chest pain and suspected myocardial infarction. We previously developed an AI-based 6- and 12-lead ECG analysis algorithm for diagnosing myocardial infarction. Based on external validation, the sensitivity, specificity, positive predictive value, and negative predictive value of this algorithm were 84.4%, 88.5%, 51.8%, and 97.5%, respectively [7]. In addition, the developed AI-ECG analysis algorithm showed high predictive power for the severity, mortality, and type of myocardial infarction (ST-segment elevation myocardial infarction [STEMI] and AMI) [7]. However, no studies have employed AI algorithms for AI-ECG analysis utilizing a multicenter prospective design for diagnosing AMI. Furthermore, the SPIRIT-AI (Standard Protocol Items: Recommendations for Interventional Trials–Artificial Intelligence) and CONSORT-AI (Consolidated Standards of Reporting Trials–Artificial Intelligence) guidelines recommend that researchers create a transparent and trustworthy model and conduct rigorous clinical trials to determine whether clinical efficacy exists [8,9]. Accordingly, we planned a prospective multicenter cohort study to validate the performance of the AI-ECG analysis algorithm for AMI detection.
METHODS
Ethics statement
This study protocol was reviewed by each institutional review board of the 18 emergency medical centers and has since been approved by the final committee. Informed consent will be obtained from either the patient or their legal representative. Additionally, individual IRB approval numbers will be included in the final report.
Trial design and setting
This prospective, multicenter, cohort study will be conducted to validate the diagnostic performance of an AI-ECG analysis algorithm for AMI among patients visiting the emergency department (ED) with acute chest pain or equivalent symptoms. Eighteen EDs in Korea, from university-level teaching hospitals, will participate in this study. These EDs receive approximately 800,000 patients annually and are capable of performing emergency cardiovascular angiography and percutaneous coronary interventions.
Participants
Inclusion criteria
The study population will include adult patients (>18 years old) visiting the ED within 24 hours of the onset of chest discomfort or those who are clinically suspected of having an AMI with equivalent symptoms. Patients who arrive at the ED 24 hours after their first chest pain, those who experience aggravated chest pain within 24 hours prior to ED arrival, or those who experience recurring symptoms within 24 hours after ED arrival will be included.
Exclusion criteria
Upon arrival at the ED, patients with out-of-hospital cardiac arrest (OHCA), those who do not provide consent to participate in the study, those with traumatic chest pain or other diagnoses that are well-differentiated from myocardial infarction (such as pneumothorax), and those transferred from another hospital with confirmed AMI will be excluded. However, patients with a sustained return of spontaneous circulation after an OHCA or those transferred with suspected AMI will be included.
Deep learning model
In this study, an advanced algorithm based on a previously reported AI-ECG model, called AiTiA-MI (Medical AI Co), will be used to diagnose AMI [7]. The only input to this algorithm is the 12-lead ECG, and the output is the probability score of AMI, expressed from 0 to 100 to the first decimal place. Among the probability scores, a cutoff threshold of 99% sensitivity and 90% specificity is designated to classify the risk as low, intermediate, and high. The raw ECG data analyzed by the AiTiA-MI algorithm is a raw data format saved in general-purpose 12-lead ECG equipment. ECG data will be collected for each patient and analyzed separately, so AI-ECG analysis results will not affect the clinical decisions made by attending doctors. Raw digital ECG data will be used as the input and recorded for 10 seconds at a 500-Hz sampling rate. This algorithm judges the level of artifacts in the incoming 12-lead ECG and does not derive analysis results for participants whose data contains artifacts that can significantly affect diagnostic performance. In addition, for missing values, if data of more than 1 second are missing for more than one lead, the results will be subject to drop out.
Study protocol
There will not be any restrictions to the enrolled patients in terms of treatment because of their participation in this study, and standard care will be provided at each emergency center based on international guidelines (Fig. 1) [3,4]. The patients as well as medical staff will be blinded to the results analyzed by the AiTiA-MI algorithm. In other words, only the data will be prospectively collected and analyzed. The AI-ECG analysis results do not affect the treatment flow of the patients. Standard care refers to the first diagnosis of STEMI, as well as the diagnosis and exclusion of non-STEMI (NSTEMI) and unstable angina based on the 0/1- or 0/3- hour rule [4]. During the initial evaluation of a patient, it is essential to describe the onset and quality of chest pain and other complaints. Physicians will apply a 0-hour (baseline) cardiac biomarker and 12-lead ECG as essential tests. The 0-hour cardiac biomarkers and 12-lead ECG are mandatory tests for participation in this study, and if one test is missing, patients will be excluded from the study. Cardiac biomarkers and 12-lead ECG used for subsequent follow-ups are not essential tests. For example, if NSTE-ACS is suspected, 1- and 3-hour follow-up biomarkers are likely to exist in many cases; however, even if they are not present, the patients will not be excluded from the study. When the physician in charge reads a standard 12-lead ECG acquired within 10 minutes of arrival of the patient, a score equal to the possibility of AMI, from 0 to 10 points (in 1-point units; 0, no chance of AMI; 10, 100% chance of AMI), will be assigned and recorded. When STEMI is suspected, emergency treatment will be administered according to hospital-specific protocols. Even if STEMI is not suspected, standard care will be administered according to the 0/1- or 0/3-hour rules. Therefore, physicians will repeatedly apply 1- or 3-hour cardiac biomarkers and an ECG, as needed. The AI-ECG algorithm will analyze the 0-hour ECG conducted after this process. Meanwhile, 1- or 3-hour follow-up ECGs, conducted based on the order of the physician, will also be used for subgroup analysis.
Data measurements
We will measure and collect the following baseline characteristic data at the time of the ED visit: age, sex, weight, body mass index (BMI), and height. The following clinical information will be collected: vital signs, Killip classification, risk factors (hypertension, diabetes, hyperlipidemia, smoking history, and family history of cardiovascular disease), coronary artery disease, MI, percutaneous intervention, coronary artery bypass graft surgery, chronic heart failure, chronic kidney disease, transient ischemic attack or stroke, peripheral artery disease, and COVID-19 vaccination (mRNA type and inoculation within 6 weeks before the index visit). Chief complaints and onset time (typical chest pain, atypical chest pain, or noncardiac chest pain), clinical risk score (HEART [history, ECG, age, risk factors, and troponin level] and GRACE [Global Registry of Acute Coronary Events] 2.0), AMI score (range, 0–10) suspected by the physician, route of ED visit, cardiologist call time, coronary angiography unit departure time from the ED, coronary angiography unit arrival time, door-to-balloon time, and intravenous thrombolytic data will also be collected [10–12]. The laboratory findings will be planned and measured as follows: 0-hour (initial tests conducted after ED admission) blood test (blood urea nitrogen, serum creatinine, hemoglobin, and high-sensitivity [hs]-troponin I or T), and 1- and 3-hour follow-up (if the follow-up was performed before invasive coronary angiography) of hs-troponin I or T, raw 0-hour ECG data, and 1- and 3-hour follow-up. Other test results and clinical information related to the following final diagnoses will be collected: initial invasive coronary angiography results after the index visit and final diagnosis in the ED and at discharge. Chest radiography, coronary computed tomography angiography, echocardiography, treadmill test, single-photon emission computed tomography, positron emission tomography, and cardiac magnetic resonance imaging will be performed during a 30-day follow-up period before or after the index visit.
Reference standard
There may be gaps in the clinical protocols in the 18 emergency centers owing to differences in hospitals and medical staff, which may affect the diagnosis of ACS. Therefore, to minimize these limitations, a multicenter study will be conducted at a tertiary-level or certified cardiovascular hospital, where treatment will be performed according to the 0/1- or 0/3-hour rule based on international guidelines [4]. In addition, each center will use hs-troponin I or T rather than conventional troponin. Tests using point-ofcare equipment will not be accepted as initial cardiac biomarkers. In addition, to establish a reference standard, two emergency medicine specialists from each center, as independent assessors, will perform final labeling for each patient to determine whether STEMI, NSTEMI, unstable angina, stable angina, or other differential diagnoses exist. The assessors will label a type of myocardial infarction. The labeling results of the two specialists will be compared, and in case of discrepancies, a third specialist will review the results. This review will be based on the latest guidelines of the American Heart Association and European Heart Society’s fourth universal definition of MI [13].
Coronary artery disease
This includes patients diagnosed with MI, unstable angina, or stable angina prior to the index visit.
Chronic kidney disease
This includes patients with glomerular filtration rate (GFR) <60 mL/min/1.73 m2 (GFR categories, G3a–G5) or markers of kidney damage. We will review the patient’s history and previous measurements to determine the duration of kidney disease; if the duration is >3 months, chronic kidney disease will be confirmed [14].
Transient ischemic attack or stroke
We will review the medical history of the patient to determine the existence of a previously diagnosed transient ischemic attack or cerebrovascular infarction according to the latest evidence [15].
Peripheral artery disease
This is a lower extremity peripheral artery disease with atherosclerotic obstruction from the aortoiliac segments to the pedal arteries [16].
Chronic heart failure
We will review the medical history of the patient to confirm the diagnosis of heart failure before the ED visit by performing a review of the electronic medical record (EMR) [17].
Outcome variables
The primary outcome is the diagnosis of an AMI during index admission. AMI includes type 1 and type 2 MIs, as defined in the fourth universal definition of myocardial infarction guidelines [13]. There may be differences in the 99th percentile upper limit of cardiac troponin levels according to the laboratory criteria of each participating hospital. In this study, the values will be based on the standards of each hospital. The secondary outcome is a 30-day MACE. MACE is defined as any cause of death, MI, stroke, target-vessel revascularization, or stent thrombosis occurring within 30 days of an index visit. Thirty days after the index visit, a telephone follow-up will be conducted and medical records (consultation, work-up, and diagnosis lists) will be checked to determine whether MACE have occurred. We will also use mortality data provided by Statistics Korea (Daejeon, Korea).
Audits of primary and secondary outcomes will be performed by three emergency medicine specialists blinded to the results of the AI-ECG analysis. These specialists will make decisions based on prospectively collected data according to the study protocol, chest pain, angina symptoms, history, and various examination records confirmed using the EMR. In case of disagreements, the principle of majority rule shall be followed.
Sample size
The sample size was calculated based on a negative predictive value of 97.5%, as established previously [7]. The sensitivity, specificity, α error, β error, and prevalence rate were set as 84%, 88%, 0.05, 0.10, and 10%, respectively. The prevalence of AMI in adults who visit the ED for nontraumatic chest pain differs in the literature and is reported to range from 4% to 17% [3,18]. The study sample size was calculated based on a 10% prevalence rate. However, based on a dropout rate of 10%, 8,814 participants will be ultimately required. This calculation is based on the “bdpv” library of R ver. 4.1.0 (R Foundation for Statistical Computing) and a corresponding study [19].
Analysis for main outcomes
We will evaluate the performance of the AI-ECG analysis algorithm for the primary and secondary outcome variables. The accuracy metrics include area under the receiver operating characteristic curve, sensitivity, specificity, positive predictive value, and negative predictive value, along with 95% confidence intervals. We will also demonstrate the performance of the AiTiA-MI algorithm using two thresholds to guide clinical decisions, with probabilities described as continuous variables. The prespecified performance criteria (low-probability group, sensitivity ≥99.0%; high-probability group, specificity ≥90.0%) are defined based on previous studies [20,21]. Regarding the primary and secondary outcomes, the actual performance of the algorithm using these thresholds will also be analyzed.
Other analyses
The scheme of the comparative analysis of the diagnostic performance of the algorithm for the primary outcome is as follows: AI-ECG analysis versus HEART/GRACE 2.0 score comparison, AI-ECG analysis versus cardiac biomarker (initial hs-troponin I or T) comparison, and AI-ECG analysis versus the subjective AMI score provided by the ED physician regarding the initial 12-lead ECG. We plan to conduct subgroup analyses according to the following items: demographics (age, sex, and BMI), risk factors, past medical history, onset of chest pain (<3 or ≥3 hours), type of chest pain, type of MI (type 1 or type 2), type of ACS (STEMI, NSTEMI, unstable angina, NSTE-ACS, or total ACS), involvement of vessel territories (if ACS was diagnosed), and type of ECG (left bundle branch block, right bundle branch block, left ventricular hypertrophy, pacemaker rhythm, or post–return of spontaneous circulation).
DISCUSSION
The significance of this study is as follows. To the best of our knowledge, this is the first study to prospectively validate an AI-based 12-lead ECG analysis algorithm for AMI diagnosis in an external multicenter environment. Myocardial biomarker test levels only rise 3 to 6 hours after the onset of AMI and reach the highest level within 12 to 24 hours [4]. In contrast, changes in the ECG appear immediately upon the onset of AMI. Therefore, AI-ECG analysis can be more accurate and faster than myocardial biomarker tests in screening for AMI in the hyperacute state, which is the golden time for the patient. The present study is expected to provide a basis for this argument. This study will also partially verify whether the diagnostic performance of the AI-based 12-lead ECG analysis algorithm for AMI is maintained even in patients with a history of MI, valvular heart disease, congenital heart disease, and/or chronic heart failure. By conducting subgroup analyses, the study will identify subgroups where the algorithm shows low diagnostic performance in detecting AMI. For example, in the case of blood cardiac biomarkers, such as hs-troponin I or T, false-positive confounders, such as renal disease and sepsis exist. With reference to this, the input data required to upgrade this algorithm in such cases will be determined, and insights will be obtained regarding whether the algorithm should be modified into a separate version rather than upgrading, thus pursuing model diversification. This study will trigger further validation and intervention research, particularly in multinational and multiethnic settings.
Notes
ETHICS STATEMENT
This study protocol was reviewed by each institutional review board of the 18 emergency medical centers and has since been approved by the final committee.
CONFLICT OF INTEREST
Kyuseok Kim is a member of the Clinical and Experimental Emergency Medicine Editorial Board. Min Sung Lee and Joon-myoung Kwon are researchers of Medical AI Co (Seoul, Korea); Joon-myoung Kwon is also the founder and stakeholder of Medical AI Co. However, they were not involved in the peer reviewer selection, evaluation, or decision process of this article. No other potential conflict of interest relevant to this article was reported.
FUNDING
None.
AUTHOR CONTRIBUTIONS
Conceptualization: KK; Formal analysis: MSL; Investigation: MSL; Methodology: all authors; Project administration: KK, JK; Resources: JK; Software: MSL, JK; Supervision: KK; Validation: KK, TGS; Writing–original draft: MSL; Writing–review & editing: all authors. All authors read and approved the final manuscript.
Acknowledgements
The full list of the ROMIAE study group: Sung Phil Chung (Department of Emergency Medicine, Yonsei University College of Medicine, Seoul, Korea), Eunah Han (Department of Emergency Medicine, Yonsei University College of Medicine, Seoul, Korea), Dong Hoon Kim (Department of Emergency Medicine, Gyeongsang National University College of Medicine, Jinju, Korea), Sung Hyuk Choi (Department of Emergency Medicine, Korea University College of Medicine, Seoul, Korea), Sung-Jun Park (Department of Emergency Medicine, Korea University College of Medicine, Seoul, Korea), Hanjin Cho (Department of Emergency Medicine, Korea University Ansan Hospital, Ansan, Korea), Sejoong Ahn (Department of Emergency Medicine, Korea University Ansan Hospital, Ansan, Korea), Mi Jin Lee (Department of Emergency Medicine, Kyungpook National University School of Medicine, Daegu, Korea), Haewon Jung (Department of Emergency Medicine, Kyungpook National University School of Medicine, Daegu, Korea), Han Sung Choi (Department of Emergency Medicine, Kyung Hee University College of Medicine, Seoul, Korea), Seok Hoon Ko (Department of Emergency Medicine, Kyung Hee University College of Medicine, Seoul, Korea), Ki Young Jeong (Department of Emergency Medicine, Kyung Hee University College of Medicine, Seoul, Korea), Yonghee Lee (Department of Emergency Medicine, CHA University School of Medicine, Bundang, Korea), Jong Eun Park (Department of Emergency Medicine, Samsung Medical Center, Seoul, Korea), Taerim Kim (Department of Emergency Medicine, Samsung Medical Center, Seoul, Korea), Heajin Chung (Department of Emergency Medicine, Soonchunhyang University College of Medicine, Asan, Korea), Won Young Kim (Department of Emergency Medicine, Asan Medical Center, Seoul, Korea), June-sung Kim (Department of Emergency Medicine, Asan Medical Center, Seoul, Korea), Young Gi Min (Department of Emergency Medicine, Ajou University School of Medicine, Suwon, Korea), Bangshill Rhee (Department of Emergency Medicine, Ajou University School of Medicine, Suwon, Korea), Chul Han (Department of Emergency Medicine, Ewha Womans University Mokdong Hospital, Seoul, Korea), Keon Kim (Department of Emergency Medicine, Ewha Womans University Mokdong Hospital, Seoul, Korea), Jaechol Yoon (Department of Emergency Medicine, Jeonbuk National University Hospital, Jeonju, Korea), So Eun Kim (Department of Emergency Medicine, Jeonbuk National University Hospital, Jeonju, Korea), Eujene Jung (Department of Emergency Medicine, Chonnam National University Hospital, Gwangju, Korea), Woo Jeong Kim (Department of Emergency Medicine, Jeju National University Hospital, Jeju, Korea), Ji Hwan Bu (Department of Emergency Medicine, Jeju National University Hospital, Jeju, Korea), Je Hyeok Oh (Department of Emergency Medicine, Chung-Ang University College of Medicine, Seoul, Korea), Chiwon Ahn (Department of Emergency Medicine, Chung-Ang University College of Medicine, Seoul, Korea), Myeong Namgung (Department of Emergency Medicine, Chung-Ang University College of Medicine, Seoul, Korea), Jeong Yeol Seo (Department of Emergency Medicine, Hallym University Medical Center, Seoul, Korea), Dong Won Kim (Department of Emergency Medicine, Hallym University Medical Center, Seoul, Korea), Tae Ho Lim (Department of Emergency Medicine, Hanyang University College of Medicine, Seoul, Korea), Yongil Cho (Department of Emergency Medicine, Hanyang University College of Medicine, Seoul, Korea), Jae Seong Kim (Department of Critical Care and Emergency Medicine, Incheon Sejong Hospital, Incheon, Korea), Seong Beom Oh (Department of Emergency Medicine, Dankook University College of Medicine, Cheonan, Korea), Jeong Min Son (Medical AI Co, Seoul, Korea), Hak Seung Lee (Medical AI Co, Seoul, Korea), Min-yeong Kim (Medical AI Co, Seoul, Korea), Nuri Shin (Medical AI Co, Seoul, Korea), Sora Kang (Medical AI Co, Seoul, Korea), Jong-Hwan Jang (Medical AI Co, Seoul, Korea), Yong-Yeon Jo (Medical AI Co, Seoul, Korea).
References
Article information Continued
Notes
Capsule Summary
What is already known
Acute myocardial infarction (AMI) is important disease in emergency medicine, and electrocardiogram (ECG) plays an important role to detect AMI. Recently artificial intelligence (AI) has been investigated to detect AMI. However, there is no large prospective observational study to confirm the usefulness of AI in AMI.
What is new in the current study
This multicenter prospective observational study would investigate the diagnostic performance of AI-ECG to detect AMI. The results might be helpful to see the usefulness of AI-ECG for the diagnosis of AMI.