A leakage-safe validation protocol per case with invariance gates: the bias of the holdout AUC under fitting-path leaks, their detection by the gates, and what validation tests see of a development-only leak
Year: 2026
Type: preprint
Abstract: Binning, weights of evidence, screens, configuration choices and calibration maps are fitted with rows that the validation of a probability-of-default model treats as new. We state a leakage-safe protocol per case: a split fixed before any fit, a training window for dated cases, and three invariance gates that fail when a fitted parameter changes under a perturbation of rows outside the training slice. We prove that the holdout estimate is unbiased when no step read the holdout. Under the protocol the AUC bias on planted data is 0.000 ± 0.002 with 95.3% coverage of the 95% interval, and its interval contains zero on 2,500-row samples of 30,000 card holders and on the product's scorecard. On the planted data a weight of evidence with holdout labels adds +0.018 and a fit on every row +0.025, with coverage falling to 66.7%. At least one gate fails for each of 15 leaks in all 100 draws, and none on the protocol. A development-only leak is detected by the AUC against development in 99.0% of repetitions at a shift of 0.5 standard deviations and by the stability index in 14.2% at a shift of one. On five loan cohorts a random split reproduces the level of the 5-year rate within 1.2 standard errors, where the declared cohort shows gaps of up to 16.2 points. Of nine predictions, three did not hold; they were written with the outputs they decide. The gates do not detect a feature built from information dated after the observation date. Repository and reproducible pipeline (MIT): https://github.com/fsantibanezleal/CAOS_Contraste .
Author: Santibáñez-Leal, Felipe
Language: English
Subfield: Accounting
Domain: Social Sciences
Open Access status: green
OpenAlex ID: W7225619873
Cites:
Cited by:
Related to: