Twelve of 21 credit-risk validation tests hold their size on 200,000 null repetitions: a measurement of size and power on generators of known truth
Year: 2026
Type: preprint
Abstract: Validation reports judge the risk of a model by statistical tests whose error rates the supervisory texts do not state. We measure the size and power of each test on generators of known truth, under its null and under planted defects, with a Monte Carlo standard error and a Wilson interval for every rate and the exact probability where one exists. A test holds its size unless its exact rejection probability or the Wilson interval of its rate lies above the level at 5 % or 1 %. Of 21 credit-risk tests, 12 hold their size on 200,000 null repetitions each and 9 do not; the largest excesses are the Jeffreys test, with an exact size of 1.38 % at the 1 % level, and the normal test over five years, which rejects 7.99 % of correct models at 5 % (Wilson interval 7.87 to 8.11 %). For 10 of the 12 the null is composite and the size is measured at one configuration. Cells of 4,000 repetitions resolve 1.00 point and show 18 holding, so the count depends on them. Tests that assume independent defaults reject a correct PD in 19.7 % of samples at an asset correlation of 10 %, the 0.10 rule of the stability index alarms in 85.0 % of unchanged samples of 100, and the European Central Bank's concentration test cannot reject at 5 % with ten grades at an initial coefficient of variation of 0.6. Of 17 power curves, 13 reach 80 % on their ladder; a planted ladder is not a real bank. Repository and reproducible pipeline (MIT): https://github.com/fsantibanezleal/CAOS_Contraste .
Author: Santibáñez-Leal, Felipe
Language: English
Subfield: Finance
Domain: Social Sciences
Open Access status: green
OpenAlex ID: W7225652905
Cites:
Cited by:
Related to: