Get started
Work

Three corrections to model-validation tests (conformal coverage, shift resilience, situation testing): the false-alarm rate each removes and the power each costs, measured on planted data

Year: 2026
Type: preprint
Abstract: The literature that recommends a validation test rarely states its size and power. We measure the error rate that three corrections remove in an open risk-model engine, running each rule and the rule it replaces on the same seeded draws of known-truth generators, with the standard error of every rate. The coverage test of a split-conformal set reads the covered count against the Beta-Binomial law of a finite calibration window; the resilience test reads the AUC drop on the worst subsample against a permutation reference; the situation-testing light takes its amber cut from a measured size and power, and the rule selects the engine's cut 0.20 on one of four design and size pairs. At equal calibration and test windows the binomial coverage test rejects an exchangeable window in 0.157 of the replications (standard error 0.006) against 0.051 for the Beta-Binomial test, whose exact size never exceeds 0.0499; a binomial test with its cut calibrated to the exact size has the same power within 0.019. With 500 evaluation rows a fixed drop of 0.05 raises a false alarm in 0.175 of shift-free replications against 0.062 for the permutation reference, which detects a shift of two standard deviations in 0.680 against 0.925. With 400 applicants the cut 0.20 raises a false alarm in 0.014 against 0.114 for the cut 0.10. Of 10 predictions, 3 are refuted: the cut 0.20 costs power at small gaps, and its size depends on the applicants and the data process. Repository and reproducible pipeline (MIT): https://github.com/fsantibanezleal/CAOS_Contraste .
DOI: 10.5281/zenodo.24053187
Language: English
Open Access status: green
OpenAlex ID: W7225653108
Cites:
Cited by:
Related to: