Work

Satisfying Model Validity for Robust Clinical Machine Learning for Time-Series

Year: 2023
Type: dissertation
Abstract: Many healthcare data are endogenously produced as a by-product of care. The chronologically recorded healthcare system interactions naturally become time-series data. Machine learning models built to infer meaningful information from these data have the ability to capture the complex dynamics of human health leading to an improvement in care without an escalation in cost. However, these models must be safe and effective in order to have an impact on healthcare. Safety and efficacy stem from models which are robust to perturbations. This thesis investigates machine learning robustness under three prominent dataset shift settings. First, this thesis identifies common design pitfalls that lead machine learning models to immediately violate internal validity. Such models will not be expected to generalise to deployment scenarios. The effects of these design decisions are highlighted and remedied. Next, internal validity violations due to dataset shift are explored. In an attempt to preserve privacy in de-identified datasets, models developed for electronic health record prediction tasks have been blinded to damaging covariate shift. In COVID-19 detection tasks using wearable data, failure to address prior probability shift leads to inflated expectations of performance. By addressing feature representation and study design, this thesis reports performance that could be reasonably expected during deployment. Finally, necessary external validity violations due to data scarcity are explored. With endogenously collected data, especially health data, it is not feasible to invest in collecting more supervised labels. Yet label-scarce datasets are abundant, whether it be for emergent disease detection, rare disease identification, or from data-scarce institutions. Pre-training using multi-task learning is discovered as a superior method to masked imputation for target-label scarce tasks. A large EHR domain transfer experiment reveals that pretraining on external datasets only confers benefits in the few-shot transfer setting. Preference is given to participate in data warehouses rather than sharing pre-trained models. This thesis illustrates how to design effective machine learning models for endogenously produced clinical time-series, how to ensure they are robust to inevitable distribution shifts, and how to share knowledge between domains. It is essential to overcome these bottlenecks to realise the potential of machine learning for clinical time-series data.
Language: English
Open Access status: green
OpenAlex ID: W7132957342
Cites: 233
Cited by: 1
Related to: 10