Year: 2022
Type: dissertation
Study design: Systematic Review
Abstract: Objectives -Missing data is a common problem during the development, evaluation, and implementation of prediction models.Although machine learning (ML) methods are often said to be capable of circumventing missing data, it is unclear how these methods are used in medical research.We aim to find out if and how well prediction model studies using machine learning report on their handling of missing data. Study design and Setting -We systematically searched the literature on published papers between 2018 and 2019 about primary studies developing and/or validating clinical prediction models using any supervised ML methodology across medical fields.From the retrieved studies information about the amount and nature (e.g., missing completely at random, potential reasons for missingness) of missing data and the way they were handled were extracted. Results -We identified 152 machine learning-based clinical prediction model studies.A substantial amount of these 152 papers did not report anything on missing data (n = 56/152).A majority (n = 96/152) reported details on the handling of missing data (e.g., methods used), though many of these (n = 46/96) did not report the amount of the missingness in the data.In these 96 papers the authors only sometimes reported possible reasons for missingness (n = 7/96) and information about missing data mechanisms (n = 8/96).The most common approach for handling missing data was deletion (n = 65/96), mostly via complete-case analysis (CCA) (n = 43/96).Very few studies used multiple imputation (n = 8/96) or built-in mechanisms such as surrogate splits (n = 7/96) that directly address missing data during the development, validation, or implementation of the prediction model.Conclusion -Though missing values are highly common in any type of medical research and certainly in the research based on routine healthcare data, a majority of the prediction model studies using machine learning does not report sufficient information on the presence and handling of missing data.Strategies in which patient data are simply omitted are unfortunately the most often used methods, even though it is generally advised against and well known that it likely causes bias and loss of analytical power in prediction model development and in the predictive accuracy estimates.Prediction model researchers should be much more aware of alternative methodologies to address missing data.Missing data is poorly handled in prediction model studies using ML What is new?Key findings › Prediction model studies that adopt machine learning (ML) methods rarely report the presence and handling of missing data.› Although many types of machine learning methods offer built-in capabilities for handling missing values, these strategies are rarely used.Instead, most ML-based prediction model studies resort to complete case analysis or mean imputation.What this adds to what was known › Missing data are often poorly handled and reported, even when adopting advanced machine learning methods for which advanced imputation procedures are available.What is the implication, and what should change now › The handling and reporting of missing data in prediction model studies should be improved.A general recommendation to avoid bias is to use multiple imputation.It is also possible to consider machine learning methods with built-in capabilities for handling missing data (e.g., decision trees with surrogate splits, use of pattern submodels, or incorporation of autoencoders).› Authors should take note of and appreciate the existing reporting guidelines (notably, TRIPOD and STROBE) when publishing ML-based prediction model studies.These guidelines offer a minimal set of reporting items that help to improve the interpretation and reproducibility of research findings.20 Chapter 2independently reviewed fifty percent of all studies respectively.Unresolved disagreements were resolved through consensus with a third reviewer (TD).All items used in the data extraction can be found in the Appendix.For the data extraction some reporting items (e.g., Item 2.1) about identifying and handling missing data from Table 1 were split up into several separate data extraction items. ResultsAfter screening, 152 eligible articles were available for the present study (Figure 1).A total of 56 (37%) prediction model studies did not report on missing data and could not be analyzed further.We included 96 (63%) studies which reported on the handling of missing data.Across the 96 studies, 46 (48%) did not include information on the amount or nature of the missing data. Presence and mechanism of missing dataPapers that reported on the amount of missing data most often (n = 31/50 [62%]) reported the overall number or frequency of missingness (e.g., the total number of patients or variables with one or more missing values).For these papers, the overall median percentage of missingness was 4.7% (IQR 1.85-28).In most other cases it was unclear how many values were missing.It was often unclear which variables exactly were missing (n = 39/50 [78%]).In 7 papers it was explicitly stated that the outcome was missing [14%].Only a small proportion of papers provided possible Health Research and Development (#8480-34001) was obtained to develop feedback procedures.UCC website: www.umuctrecht.nl/ucc(in Dutch).
DOI: 10.33540/1375
Author: Nijman, Steven Willem Joost
Institutions: Utrecht University, University Medical Center Utrecht
Language: English
Subfield: Statistics and Probability
Field: Mathematics
Domain: Physical Sciences
Open Access status: gold
Awards: G6019094550, G6074549254
OpenAlex ID: W4281778036