Work

Transportability, calibration, and alert burden of an hourly model for documented continuous vasopressor initiation: a retrospective three-database study

Year: 2026
Type: article
Abstract: Background Clinical prediction models may retain discrimination across settings while losing probability calibration or a workable alert policy. We evaluated this distinction for hourly prediction of documented continuous vasopressor initiation. Methods We used MIMIC-IV v3.1, eICU-CRD v2.0, and SICdb v1.0.8. Untreated adult stays contributed hourly landmarks from ICU hour 6 through 24 while still under observation. The outcome was first initiation within six hours and before ICU/unit exit; exit without initiation remained an operational non-event. Histogram-based gradient boosting (HGB) and logistic regression were fitted and selected in patient-disjoint MIMIC-IV reference-period groups. Evaluation used a separate MIMIC-IV group and two external databases. Source observations were range-checked before predictor summarization; SICdb hourly aggregates were withheld until the source hour had ended. These retrospective timing rules did not verify availability in live clinical information systems. Results Evaluation included 165,204 landmarks from 9,534 MIMIC-IV reference-period test stays; 2,745,224 landmarks from 168,636 eICU-CRD stays; 58,756 landmarks from 3,997 SICdb stays. HGB AUROCs were 0.770, 0.849, 0.695, respectively. Across 100 identifier-disjoint local-update splits, median Brier skill relative to a calibration-subset constant forecast was −0.168, −0.005, 0.023 before updating and 0.019, 0.025, 0.028 after full logistic recalibration. With thresholds targeting five raw alerts per 100 calibration rows, event detection after six-hour suppression was 0.406, 0.510, 0.275; false emitted alerts were 1.64, 1.52, 1.51 per 100 evaluation rows. Thresholds targeting 80% raw landmark sensitivity detected 0.885, 0.873, 0.883 of eligible events after suppression, while 63.8%, 50.7%, 86.9% of event-free stays received at least one false alert. Conclusions Discrimination transferred unevenly, while calibration and alert performance remained setting-dependent. The evaluated model and alert policies are not ready for clinical deployment. Offline assessment against prespecified clinical criteria should precede any prospective silent evaluation.
Language: English
Open Access status: gold
Estimated APC paid: $3,801
OpenAlex ID: W7226002930
FWCI:
Cites: 27
Cited by:
Related to: 10
Parts: 1