Year: 2023
Type: article
Abstract: Rapid risk stratification for patients presenting with chest pain to the emergency department (ED) improves efficiency of care and patient outcomes.1 While the care of patients presenting with chest pain has evolved, the initial evaluation still requires serial measurement of high-sensitivity cardiac troponin (hs-cTn) to rule out an acute coronary syndrome (ACS) event. Over three-quarters of initial hs-cTn measurements are below the upper limit of normal reference values, mostly from patients who would be considered lower risk for ACS and discharged after this reading.2 However, these low-risk cases have not been the focus of commonly used prognostic scores that define a person’s risk for a subsequent myocardial infarction (MI) or death.3 For instance, an accepted rule-out method, the European Society of Cardiology (ESC) 0/1-hour algorithm, has a high negative predictive value (NPV) but still requires serial assessments of hs-cTn.4,5 Machine learning can fill this gap by defining a triage strategy based on initial patient characteristics.6 Fernandez-Cisnal et al. report the development and validation of a machine learning model for multi-dimensional data in the electronic health record (EHR) can assist in triaging patients with chest pain after a single hs-cTn measurement, identifying those at low risk for subsequent MI and death.7 This study demonstrates how to apply machine learning to solve a clinical problem by identifying key clinical features for training, choosing the best-performing model, and demonstrating applicability to a broader population. They focused on low-risk chest pain, including those with an initial hs-cTn level below the upper limit of normal and without electrocardiogram (ECG) evidence of ischaemia or arrhythmias. The model used 15 commonly acquired patient characteristics in the ED in addition to the initial hs-cTn, including age, sex, diabetes, dyslipidaemia, previous MI, vital signs, chest pain characteristics, ischaemic ECG changes, and creatinine. They trained several machine learning models on a retrospective cohort of 4075 patients who had presented with chest pain to the ED at a single hospital in Spain to predict either MI or death 90 days from clinical presentation. They externally validated their model in the Advantageous Predictors of Acute Coronary Syndromes Evaluation (APACE) Study, which followed 3609 patients with chest pain presenting to 13 EDs in Switzerland, Czech Republic, Poland, and Spain for over 5 years for recurrent symptoms and cardiovascular events. At a sensitivity of 96.9% and NPV of 99.8%, the model surpassed ESC 0/1 performance and predicted a safe ‘discharge’ of 36.7% of patients after the initial hs-cTn reading. Of those ‘discharged’, 2.2 patients per 1000 had a missed MI or death at 90 days. The model was also superior to using only a single hs-cTn below the upper reference limit, which corresponded to a discharge recommendation for 27.1% of patients, while 3.1 of these patients per 1000 had a missed MI or death at 90 days. Algorithms for safe discharge of low-risk patients are essential to reduce ED wait times but require a thorough evaluation for patient safety. A single hs-cTn below the upper limit of normal has been shown to safely rule out at least 20–50% of patients presenting with chest pain for over 2 h and a non-ischaemic ECG.8 Despite the theoretical efficacy, the concern about missing higher-risk individuals with borderline hs-cTn, as evidenced by an NPV of <99% in an American diagnostic study, has limited its clinical use.9 Other traditional algorithms for ruling out ACS, such as the Global Registry of Acute Coronary Events (GRACE) score and History, ECG, Age, Risk factors, and Troponin (HEART) score, were not developed or evaluated for lower-risk chest pain. Therefore, models exclusively developed for this application hold promise. In addition to Fernandez-Cisnal et al.’s work, another recent study demonstrated that combining clinical characteristics with a single hs-cTn allowed for rapid risk stratification. The Collaboration for the Diagnosis and Evalaution of Acute Coronary Syndrome (CoDE-ACS) score used another machine learning approach, gradient boosting, with excellent performance for predicting MI risk in those presenting with chest pain (Area Under the Reciever Operating Curve (AUROC) of 95%, NPV 99.7%). This was based on the initial hs-cTn and 12 patient characteristics and validated in seven pooled cohorts not used in the development of the algorithm.10 The CoDE-ACS further enables additional conclusions from the current study: (i) combining a single presenting hs-cTn measurement with clinical features at presentation is prognostic, and (ii) machine learning methods can enhance risk stratification and augment healthcare decision-making, especially when combining multiple heterogeneous data streams. While Fernandez-Cisnal et al.’s work and CoDE-ACS exemplify the value of machine learning–based tools, the development of models using these approaches requires careful consideration of the data source, the choice of model, and an evaluation outside the development population that confirms its generalizability.11 When considering a data source, models are best developed in carefully curated populations where included patients represent the corresponding real-world populations, all candidate covariates are consistently measured regardless of the clinical presentation, and outcomes can be consistently measured. Moreover, machine learning methods require such data sets to be large. When evaluating the EHR for such use, it is essential to assess whether the clinical setting in the EHR approximates the ideal model development population.12 For chest pain evaluations, testing is standardized, but the definition of the condition often requires evaluation of documentation of the presenting symptoms, especially as the diagnosis code is insufficient. After the data source, the next decision point is the choice of the model. Evaluating potential predictors against an outcome directly is considered ‘supervised’ machine learning. This contrasts with ‘unsupervised’ machine learning, where the model does not depend on the occurrence of the outcomes but instead focuses on identifying patterns within the data. These patterns can define new population subsets with shared characteristics and outcomes. The present study uses supervised learning and, therefore, the adjudication of cases and controls for training has an outsized influence. For example, training a model on healthy controls could lead to higher performance in development but fail when applied to real-world patients with comorbidities that are predictive of the outcome but were not available or considered during training.13 Fernandez-Cisnal et al.’s model performance decreased from training to external validation, demonstrating the challenges of maintaining high model performance when testing in a new patient population. This study compared three supervised models: logistic regression, neural networks, and gradient boosting, which varied in performance. Multivariate logistic regression, a standard approach for developing prognostic scores, identifies linear relationships between covariates and the outcome. However, more complex effects, such as those arising from non-linear associations between covariate and outcome and complex covariate–covariate interactions, are challenging to capture. Neural networks, on the other hand, consist of layers of interconnected and weighted nodes, each representing a regression equation learning from data as it is processed through each layer.11 The successful development of neural networks, however, relies on large numbers of training cases and controls with minimal missingness, which can be challenging for a specific clinical cohort. The third and best-performing algorithm in the current study, a gradient boosting model, typically combines the results of many smaller models called decision trees that leverage different sets of characteristics for prediction. Methods vary on how the trees are combined, but in gradient boosting, the model focuses on reducing the error residuals in the ‘weakest’ decision trees to improve accuracy in the toughest discrimination cases.11 Their learning method allows for minimal pre-processing of mixed data sets, including categorical and continuous data, which are common in the EHR. In addition, their structure allows for identifying non-linear patterns in complex data and interpretability, often lacking in deep learning methods such as neural networks. Further, ensemble methods require less data than neural networks, which is beneficial in medical studies when cases are few. Similar to this study, others have shown gradient boosting outperforming other machine learning methods in risk stratification.14,15 Overall, gradient boosting models offer a favourable balance between handling complex phenotypes and interpretability. Finally, the generalizability of the model with maintained high sensitivity and NPV is essential for clinical application in the ED. High performance in an external validation cohort can indicate good generalizability. This study model had a high AUROC of 81% on their external validation cohort, APACE, though their training set had an AUROC of 92%, suggesting the model did not perfectly learn the underlying association of the covariate and outcomes and captured unmeasured differences between the development and validation cohorts. This contrasts with CoDE-ACS, which demonstrated more consistent performance in external populations and suggests that model development with similar inputs can achieve different generalizability with different development populations and approaches. Nevertheless, the works by Fernandez-Cisnal et al. and the Code-ACS team are important contributions to clinically useful machine learning algorithms that can enable better care and patient outcomes. They are ideal models for further evaluation in randomized clinical trials and eventual clinical use. Therefore, machine learning algorithms can augment personalized risk assessments from existing clinical data, enhancing our ability to provide the best care to patients while also improving care efficiency. Achieving such a future, however, would require keen attention to rigorous development and validation of algorithms before their eventual clinical use. None declared. No new data were generated or analysed in support of this research.
Authors: Phyllis M Thangaraj, Rohan Khera
Institutions: Yale University, Yale New Haven Hospital
Language: English
Subfield: Health Information Management
Field: Health Professions
Domain: Health Sciences
Open Access status: bronze
OpenAlex ID: W4387327395