The Model That Wins on AUC Is Not the Model That Saves the Bank Money: A Cost-Sensitive Re-Evaluation of Credit Default Prediction Classifiers
Year: 2026
Type: preprint
Abstract: Published benchmarks for credit-card default prediction almost universally rank classifiers by rank-based or threshold-free metrics such as AUC, F1, or accuracy, implicitly treating a missed default and a wrongly rejected good customer as equally costly errors. In practice, a bank's loss from an undetected default (loss given default on the full credit exposure) is an order of magnitude larger than its opportunity loss from rejecting a creditworthy applicant (one month of foregone interest margin), so a model ranking built on symmetric-cost metrics need not reflect which model actually minimizes the bank's expected loss. We revisit the UCI "Default of Credit Card Clients" benchmark (30,000 accounts, Taiwan, 2005) - a dataset underlying dozens of published single-split classifier comparisons - using a paired, Optuna-tuned, repeated stratified cross-validation protocol (5-fold x 6 repeats, 30 paired folds) across Logistic Regression, Random Forest, XGBoost, and LightGBM. Each model is evaluated twice: once under the conventional 0.5 decision threshold with AUC/F1/MCC, and once under a nested, leakage-free, cost-optimized decision threshold derived from an exposure-weighted cost matrix built directly from each account's own credit limit. Friedman and post-hoc Nemenyi tests show LightGBM is the significantly best-ranked model by AUC (p<0.001 vs. Random Forest), yet Random Forest attains the lowest mean expected cost (1.24% of portfolio exposure), with no statistically significant cost difference among the three tree ensembles after correction for multiple comparisons - the model ranking flips depending on which metric is used to read the same cross-validation folds. A far larger and more consistent effect, however, is threshold calibration itself: replacing the default 0.5 cutoff with a cost-optimized threshold cuts expected cost by 86-88% for every model tested (Wilcoxon p<10^-8), an effect that dwarfs the 1-6 percentage-point AUC spread the literature treats as the interesting comparison. These results suggest that for applied credit-scoring work, calibrating the decision threshold to the institution's own cost structure delivers more value than choosing among competitive ensemble algorithms, and that AUC-based leaderboards can be actively misleading about which model a bank should deploy.
Author: Prakasa, Nezar Abdilah
Language: English
Subfield: Accounting
Domain: Social Sciences
Open Access status: green
OpenAlex ID: W7235274882
