Year: 2021
Type: dissertation
Study design: Observational Study
Abstract: The aim of this thesis is to investigate the ability of ML models to make clinically useful predictions about individual patients. More specifically, it aims to address whether the combination of clinical predictors with multimodal biomarkers can increase model performance in comparison to models that only use a single data modality (i.e., only clinical or neuroimaging data). However, before these questions can be answered, psychiatric ML practices need to be standardised to ensure that differences between models are attributable to specific data modalities and or their combination instead of natural variation resulting from small samples, data leakage, diverse cross-validation schemes, and experimenter degrees of freedom. Therefore, the first half of this thesis is devoted to addressing these fundamental methodological issues in the field. First, Chapter 1 provides a general background on the heterogeneity of psychiatric disorders, an overview of group-level statistics, their limitations for modelling individual differences, the implications this has for clinical translation, and the capability of machine learning methods to assist in overcoming this problem. Chapter 2 proceeds to lay a foundation for the standardisation of methodological best practice, covering issues of model training and evaluation and clinical application. Specifically, I review literature highlighting the effects of sample size on model performance, and the counterintuitive inverse relationship between decreasing sample size and increasing model performance. In addition, I explore the issue of cross-validation scheme selection and its influence on the variability and bias of model performance estimates. In particular, I discuss commonly used schemes and their specific use cases in psychiatric studies, dependent on factors such as sample size, the presence of multiple data collection sites, and model selection and hyperparameter tuning. More specifically, I considered how CV scheme choice and structure, alongside the correct specification of pipeline architectures within these schemes has implications for data leakage and overfitting, two widespread problems in the field50,66,68-70. Finally, motivated by the observation that many editors and funding bodies lacked a measured way to interpret the maturity and impact of predictive psychiatric studies, I devised a framework and scoring system to measure the maturity of applied machine learning publications. Next, Chapter 3 proceeds to build a case for the importance of clinical context in model construction, paying specific attention to the overlooked significance of model calibration and its implications for clinical decision making. Here, I build the case for the importance of model probability calibration and demonstrate the properties of reliability curves for commonly used models in psychiatric machine learning. Given the counterintuitive inverse relationship between sample size and model performance, Chapter 4 attempts to quantify where in the cross-validation cycle performance estimates are most susceptible to variation and performance overestimation. Here, we demonstrate that the risk of misestimating model performance predominantly arises from small test set sizes. Moreover, we demonstrate through the use of dummy classifiers that this effect is not specific to any data modality or psychiatric phenotype but is a generalisable statistical effect applicable to all machine learning applications. To overcome this problem, we suggest that future studies require larger testing samples, or conduct further external testing across diverse data collection sites. Informed by these findings, Chapter 5 builds a case for a cohesive AI ecosystem, wherein issues of data collection, model construction, evaluation, and clinical translation are considered. In particular, an extensive and open-sourced model evaluation framework is proposed paying particular attention to issues of heterogeneity and sample size. In addition, new methodological techniques are introduced that may help to overcome these issues in future studies. Once this methodological foundation is laid, the second half of this thesis progresses onto addressing the benefit of introducing multimodal biomarkers into model construction alongside more traditional clinical measures (e.g., symptom items and measures of cognitive function). To this end, Chapter 6 assesses whether it is possible to predict rehospitalization in an inpatient cohort of major depressive patients using baseline clinical, structural imaging, blood-biomarker, polygenic risk, bioelectrical impedance, and electrocardiography predictors. Here, we find that a multimodal panel composed of structural imaging, blood-biomarker, clinical, medication type, and sleep quality predictors performs best. Following, Chapter 7 uses a combination of routine clinical data alongside measures of polygenic risk to predict lithium response in bipolar patients. This paper progresses beyond just using all combined predictors in a supervised ML models feature space, and instead, also uses polygenic risk (PRS) loadings for major depression and schizophrenia to stratify patients prior to model training. This approach affords the parsing of biological heterogeneity before proceeding to supervised model training with routine clinical data. Here, we found that a model stratified with a standardised meta-PRS, combining scores for both major depressive disorder and schizophrenia best predicted continuous and binary lithium response. Finally, Chapter 8 concludes the work of this thesis with a general discussion of findings and a set of new considerations for the field. Specifically, we discuss the clinical and biological heterogeneity within psychiatric traits as well as different techniques and modelling approaches that could be used to parse this heterogeneity in future works. Moreover, we address some of the methodological limitations in these studies that have led to failures in replication, and how these problems could be avoided moving forward. We end this thesis by considering the future role of technology in psychiatry, and how future investments into innovative technology may lead to paradigm shifts in the way we diagnose and treat mental health disorders
Author: Cearns, Michah Linton Douglas
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
Awards: G1116979012, G179444772, G235093469, G238323396, G2831939486, Die bidirektionale Analyse der Beziehung zwischen subklinischer Arteriosklerose und der Depression (BiDirect), G6335375501, G648958512, G6733821374, Analyse der Beziehung zwischen subklinischer Arteriosklerose und der Depression (BiDirect) (Anschlussvorhaben), Geschlechtsunterschiede im alternden Gehirn - Muster, Mechanismen und Auswirkungen, G8819810260, G961545024
OpenAlex ID: W7049394320