Performance evaluation of predictive AI models to support medical decisions: Overview and guidance
Year: 2024
Type: preprint
Abstract: A myriad of measures to illustrate performance of predictive artificial intelligence (AI) models have been proposed in the literature. Selecting appropriate performance measures is essential for predictive AI models that are developed to be used in medical practice, because poorly performing models may harm patients and lead to increased costs. We aim to assess the merits of classic and contemporary performance measures when validating predictive AI models for use in medical practice. We focus on models with a binary outcome. We discuss 32 performance measures covering five performance domains (discrimination, calibration, overall, classification, and clinical utility) along with accompanying graphical assessments. The first four domains cover statistical performance, the fifth domain covers decision-analytic performance. We explain why two key characteristics are important when selecting which performance measures to assess: (1) whether the measure's expected value is optimized when it is calculated using the correct probabilities (i.e., a "proper" measure), and (2) whether they reflect either purely statistical performance or decision-analytic performance by properly considering misclassification costs. Seventeen measures exhibit both characteristics, fourteen measures exhibited one characteristic, and one measure possessed neither characteristic (the F1 measure). All classification measures (such as classification accuracy and F1) are improper for clinically relevant decision thresholds other than 0.5 or the prevalence. We recommend the following measures and plots as essential to report: AUROC, calibration plot, a clinical utility measure such as net benefit with decision curve analysis, and a plot with probability distributions per outcome category.
Source: arXiv (Cornell University)
Authors: Van Calster, Ben, Collins, Gary S.; id_orcid 0000-0002-2772-2316, Vickers, Andrew J., Wynants, Laure, Kerr, Kathleen F., Barreñada, Lasai, Varoquaux, Gael, Singh, Karandeep, Moons, Karel G. M., Hernandez-Boussard, Tina, Timmerman, Dirk, Mclernon, David J., Smeden, Maarten Van, Steyerberg, Ewout W.
Institutions: Leiden University Medical Center, KU Leuven, University of Oxford, Memorial Sloan Kettering Cancer Center, University of Washington, Centre Inria de Saclay, SODA: Méthodes computationnelles et mathématiques pour comprendre la société et la santé à partir de données, UC San Diego Health System, University of California San Diego, Universitair Ziekenhuis Leuven, University of Aberdeen, Leiden University
Language: English
Subfield: Health Informatics
Field: Medicine
Domain: Health Sciences
Open Access status: green
Funders: Cancer Research UK, National Institute for Health and Care Research, Department of Health and Social Care, Agence Nationale de la Recherche, ZonMw, Fonds Wetenschappelijk Onderzoek, KU Leuven, Kom op tegen Kanker, Vlaamse regering, National Institutes of Health, Stand Up To Cancer, Medical Research Council, Engineering and Physical Sciences Research Council, National Cancer Institute, National Institute of Diabetes and Digestive and Kidney Diseases, U.S. National Library of Medicine, National Center for Advancing Translational Sciences
Awards: G1190173965, Optimising ovarian tumour management: a multicentre study of the short and long term effects of the surgical versus conservative management of ovarian masses., Systematic Reviews and Meta-Analysis of Prognosis Studies (REVAMP): development of core methods, reporting guidelines and a methodology handbook, G1969776114, Sample Size guidance for developing and validating reliable and fair AI PREDICTion models in healthcare (SS-PREDICT), Stanford Center for Clinical & Translational Education and Research (Spectrum), G3450463239, Taking ovarian cancer diagnosis to the next level: differential diagnosis of malignancy type, and added value of advanced data sources such as MRI, proteomics, circulating tumour DNA and circulating tumour cells, G4771589218, Statistical and AI based Methods for Advanced Clinical Trial CHallenges in Digital Health, G6019094550, G6074549254, G646799117, G6640297445, Advancing Knowledge Discovery for Postoperative Pain Management, The ADNEX risk model for ovarian cancer diagnosis: improving performance across different hospitals and examiners, Dendritic cell immunotherapy in serous ovarian cancer., Cancer Center Support Grant, Novel Kidney Injury Tools in Deceased Organ Donation to Predict Graft Outcomes, Cardiac sURgery anesthesia Best practices to reduce Acute Kidney Injury (CURB-AKI)
OpenAlex ID: W4405434087