How calibration fails when prediction models cross hospitals: logistic regression versus tree-based models for septic shock mortality
Year: 2026
Type: preprint
Abstract: Background Clinical prediction models developed at single institutions rarely undergo rigorous external validation. Whether model calibration, the agreement between predicted and observed event rates, generalizes across populations with substantially different baseline risk remains poorly characterized. Methods We developed prediction models in MIMIC-IV (n = 17,516) and externally validated them in eICU-CRD (n = 2,819) using a pragmatic septic shock definition (suspected infection, organ dysfunction, and vasopressor use). Logistic regression (LR), random forest (RF), and XGBoost were compared on native probabilities using 72 predictors. We assessed discrimination (AUC), calibration (Brier score), and calibratability with small local samples (intercept correction and Platt scaling). Results Internal AUC ranking did not predict external performance. LR ranked second internally (0.820) but achieved the highest external AUC (0.813 vs. XGBoost 0.800, RF 0.759). LR's calibration slope stayed near 1.0, and its external error was a pure location shift that 50 local patients sufficed to correct. XGBoost's raw Brier of 0.185 reflected a coincidental match between its class-weight-centered predictions and eICU's 42.4% mortality, and its uncorrected calibration was prevalence-dependent in simulation. RF's error was a compression of predicted risks that a single intercept could not repair; only full recalibration reduced its Brier score (0.211 to 0.196). Conclusions When local recalibration data are scarce, the structure of calibration failure should guide model selection. LR's failure is visible and cheap to repair, XGBoost's is prevalence-dependent and may be invisible without local data, and RF's requires full recalibration rather than a single intercept.
Source: Research Square
Institutions: Xijing Hospital, Air Force Medical University
Language: English
Field: Medicine
Domain: Health Sciences
Open Access status: green
OpenAlex ID: W7225907720
