Benchmark record: per-replicate predictions and goodness-of-fit results with the true risk known (Tabular foundation models are not a shortcut to calibrated clinical risk prediction)
Year: 2026
Type: dataset
Abstract: The per-replicate predictions and test results behind the article "Tabular foundation models are not a shortcut to calibrated clinical risk prediction: a pre-registered simulation against the true risk" by Ebrahim Khaled Ebrahim. The study was pre-registered on OSF Registries (doi:10.17605/OSF.IO/8HMTZ); this record is the open benchmark planned in Section 14 of the protocol, so that a new calibration method or test can be run on exactly the same predictions, with the true risk known. Simulation. For 58 scenarios (four true-risk shapes at event rates of 3, 10 and 30 % and development sizes of 200 to 3,000, and five stress families) and 1,000 random designs: for every evaluation record, the true risk, the outcome, the covariates, and the prediction and exact-null outcome of every arm (logistic regression, Firth logistic regression with intercept correction, ridge, GAM, LightGBM with and without Platt scaling, TabPFN-3.5, TabICLv2, Mitra-v2, and side arms), with the recalibration ladder; every row of the goodness-of-fit layer (ebrahim.gof 2.10.0) on the full evaluation sets and on nested subsamples of about 50 and 200 events; the oracle size study; and the true-risk and empirical calibration estimands of every fit. Real data. Per-replicate predictions, test results and estimands for the Sepsis Survival Minimal Clinical Records and the PhysioNet/CinC Challenge 2012, whose licences allow it. For the International Stroke Trial and CRASH-2, only aggregate results are included; no row-level, per-patient, per-subsample or per-hospital result is deposited. The CRASH-2 results are published with the permission of the trial's Chief Investigators. Also included: the tidy analysis tables, every summary table and figure of the article and its supplement, the computing-cost tables, the design tables, the registered protocol with its deviations log, a loader (benchmark_loader.py) and a data dictionary. The code that produced the record is archived separately (doi:10.5281/zenodo.23737378). Licence: CC BY 4.0.
Author: Ebrahim, Ebrahim Khaled
Institution: Alexandria University
Language: English
Subfield: Statistics and Probability
Field: Mathematics
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7224102129