Real Beats, Below the Margins: A Pre-registered Benchmark in Which Tuned XGBoost and LightGBM Edge the sklearn Credit Champion by Less Than the Frozen Winning Lines, and the Calibration Ranking Is Not the Discrimination Ranking
Year: 2026
Type: preprint
Abstract: When a credit team considers replacing a working scorecard library, the question is rarely asked under controlled conditions: the challenger is usually tuned, the incumbent usually is not, and the comparison is scored on one ranking metric. This study froze the question before measuring it. Under a pre-registration locked before any measured run, XGBoost 3.4.1 and LightGBM 4.7.0 were benchmarked against the published sklearn gradient-boosting champion of a companion study (credit-risk-lab) on that study's own leakage-safe protocol: the UCI default of credit card clients data (30,000 accounts), a stratified 70/30 split at seed 42 (21,000 training rows, 9,000 test rows), preprocessing fit on training rows only, and every arm scored on the identical preprocessed matrix. Each tuned arm received exactly 24 random-search trials under the same stratified 3-fold cross-validation on the training rows, with selection by mean cross-validation AUC and the test split scored exactly once per final model. A sixth arm, the champion's own library given the same 24-trial budget, separated the library claim from the tuning claim. The reproduction gate passed exactly: the control arm reproduced the published champion figures to the last printed digit (AUC 0.7810, KS 0.4196, Brier 0.1355, delta 0.0000 on all three). H1 is INCONCLUSIVE: tuned XGBoost beat the champion on AUC, 0.7822 against 0.7810, a delta of +0.0012, but the frozen margin for a confirmed beat was +0.005. H2 is INCONCLUSIVE: tuned LightGBM beat the champion on KS, 0.4239 against 0.4196, a delta of +0.0043, against a frozen margin of +0.010. H3 is CONFIRMED: the calibration ranking is not the discrimination ranking; by AUC the order is XGBoost, LightGBM, champion, and by Brier it is LightGBM, XGBoost, champion, so the AUC winner is not the calibration winner. With equal tuning budgets the three tuned libraries land within 0.0010 AUC of each other (0.7812 to 0.7822), which is the study's central measurement: on this protocol the champion's deficit to its tuned rivals is a tuning deficit, not a library deficit. Library defaults tell the opposite story to tuning: XGBoost at defaults trails the champion badly (AUC 0.7598), while LightGBM at defaults already beats the champion on KS (0.4237 against 0.4196). Which library wins has no answer until the tuning budget is named, and a swap chosen on the ranking metric alone would have picked the worse-calibrated of the two challengers.
Author: Ushno, Jesun Ahmad
Language: English
Subfield: Accounting
Domain: Social Sciences
Open Access status: green
OpenAlex ID: W7236065888
