Work

Bayesian Prediction of Future Empirical ROC Curves and AUC

Year: 2026
Type: preprint
Abstract: The measured performance of a classifier can change when it is evaluated on a new test sample, even when the classifier itself remains fixed. We propose DiP-ROC (Dirichlet predictive ROC), a Bayesian framework for predicting this variation from a single test sample. It predicts the receiver operating characteristic (ROC) curve and its area under the curve (AUC) that a new sample from the same populations could produce, with the same numbers of positive and negative observations. DiP-ROC groups prediction scores into bins defined by a fixed threshold grid and uses a Dirichlet--multinomial model for each class. Together, these models describe the joint uncertainty in the future ROC points and their corresponding grid-based AUC. They preserve the dependence between ROC points, which are calculated from the same observations. They also account for both uncertainty about the score distributions and the variation caused by drawing a new test sample. The resulting summaries are simultaneous predictive bands for all ROC points on the grid and predictive intervals for the AUC. Prior information can come from an earlier evaluation of the same classifier. This information may be a reported AUC value or two score histograms, one for each class. Experiments on synthetic data and nine real-world datasets show how prior strength and class counts affect the reliability and precision of predictions for a future evaluation. All code is available on GitHub to support reproducibility: \url{https://github.com/ocaelen/bayesian_roc}.
Language: English
Open Access status: green
OpenAlex ID: W7235029883
Cites: 18
Cited by:
Related to: 10