Risk Stratification for 7-Day Mortality After ICU Admission Using General-Purpose Large Language Models and Structured Electronic Health Records: Predictive Performance and Reasoning Alignment Across Three Cohorts
Year: 2026
Type: preprint
Abstract: Background Early risk stratification after intensive care unit (ICU) admission may inform treatment intensity, goals-of-care discussions, and resource allocation. Developing and maintaining supervised machine learning models requires large labelled cohorts and technical infrastructure that may not be available in all settings. General-purpose large language models (LLMs) can generate risk estimates with human-readable rationales, but their performance and reasoning quality across independent healthcare systems remain uncertain. Methods We analysed three ICU cohorts: eICU-CRD (208 US hospitals), MIMIC-IV (US), and a Korean tertiary-hospital subset of K-MIMIC (Korea). Seven-day mortality after ICU admission was predicted from routinely collected demographic, laboratory, and vital sign variables recorded during the first 24 hours. Six supervised machine learning and deep learning models were developed in eICU-CRD and externally validated in MIMIC-IV and K-MIMIC. GPT-5 and Gemini-2.5-Pro were evaluated with four prompting strategies in 2,000-stay samples per cohort, without cohort-specific training or task-specific fine-tuning. Clinical concepts mentioned in GPT-5 rationales were compared with patient-level predictor importance from an XGBoost reference model; two clinicians rated 150 rationales for clinical trustworthiness. Results Analyses included 124,020, 84,926, and 6,673 ICU stays. Among supervised models, the highest area under the receiver operating characteristic curve (AUROC) was 0.936 for XGBoost in the held-out eICU-CRD set, 0.858 for the multilayer perceptron in MIMIC-IV, and 0.957 for XGBoost in K-MIMIC. In the corresponding 2,000-stay LLM evaluation samples, GPT-5 achieved AUROCs of 0.925, 0.822, and 0.926, respectively, but generally had a lower area under the precision–recall curve (AUPRC) and, at evaluated operating thresholds, lower specificity. Overlap with the five most influential concepts was partial (precision, 0.47; recall, 0.55), but mentioned concepts had greater mean patient-level importance than nonmentioned concepts in 98.1% of patients (P < 0.001). Clinician-rated trustworthiness averaged 4.03 ± 0.63 on a 5-point scale, with ratings within one point in 83% of cases. Conclusions Supervised models generally achieved the highest discrimination for 7-day mortality after ICU admission. GPT-5 achieved high discrimination without cohort-specific training or task-specific fine-tuning, and its patient-specific rationales preferentially referenced influential predictors and received favorable clinician ratings. Prospective evaluation of reasoning faithfulness and clinical utility is required before clinical use.
Source: Research Square
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7234752088
