We want them to trust us, but can we trust them? A retrospective programmatic data analysis to assess patterns and outcomes of hospitalist agreement with AI-based mortality risk prediction (Preprint)
Year: 2026
Type: preprint
Study design: Observational Study
Abstract: BACKGROUND Machine learning can enhance mortality risk prediction within clinical decision support systems (CDSS), but its impact depends on whether clinicians trust and act on model predictions. Across our health system, we implemented a deep‑learning-based CDSS to identify inpatients at high risk of mortality and prompt goals‑of‑care discussions (GOCDs). Hospitalists receiving alerts could indicate agreement or disagreement. OBJECTIVE The objective of our study was to assess what factors were associated with clinician disagreement with the model, and whether incorporating clinician agreement improved 30-day mortality predictions and subsequent clinical action. METHODS We used penalized logistic regression followed by mixed‑effects models to identify features associated with clinician disagreement with the alert. Next, we compared two strategies for predicting 30‑day mortality among eligible encounters: utilizing a higher score threshold (≥50% rather than ≥25%) versus restricting the “high risk” designation to encounters where clinicians agreed with the alert. We then used mixed‑effects and inverse-probability‑weighted models to evaluate the impact of agreement on three post‑alert actions: GOCD documentation, palliative care referral, and code status change. RESULTS Among 1,830 alerts with clinician responses between December of 2020 (the program launch) and December of 2024, 15.7% indicated disagreement. Notable factors associated with disagreement included younger patient age, lower severity of illness, and certain diagnoses (e.g., COVID-19). Both the clinician agreement strategy and the 50% score threshold yielded modest discrimination for 30‑day mortality (AUC-ROC ≈ 0.55-0.57). Compared with the higher threshold strategy, clinician agreement provided higher sensitivity but lower specificity and overall accuracy. Clinician agreement was also associated with substantially higher rates of GOCD documentation, palliative care referral, and code status change. CONCLUSIONS In this high‑risk population, hospitalist judgment did not meaningfully improve mortality prediction and was comparable to increasing the alert score threshold to 50%, but it did identify more patients who later died, which may be preferrable in this context. Moreover, clinician agreement with alerts was critical for prompting subsequent clinical actions. CLINICALTRIAL N/A
Language: English
Field: Medicine
Domain: Health Sciences
Open Access status: green
OpenAlex ID: W7167664165