Year: 2025
Type: article
Abstract: Cross-validation is a widely used technique for evaluating the performance of prediction models, ranging from simple binary classification to complex precision medicine strategies. It helps correct for optimism bias in error estimates, which can be significant for models built using complex statistical learning algorithms. However, since the cross-validation estimate is a random value dependent on observed data, it is essential to accurately quantify the uncertainty associated with the estimate. This is especially important when comparing the performance of two models using cross-validation, as one must determine whether differences in estimated error are due to chance. Although various methods have been developed to make inferences on cross-validation estimates, they often have many limitations, such as requiring stringent model assumptions. This paper proposes a fast bootstrap method that quickly estimates the standard error of the cross-validation estimate and produces valid confidence intervals for a population parameter measuring average model performance. Our method overcomes the computational challenges inherent in bootstrapping a cross-validation estimate by estimating the variance component within a random-effects model. It is also as flexible as the cross-validation procedure itself. To showcase the effectiveness of our approach, we conducted comprehensive simulations and real-data analysis across two applications.
Source: The Annals of Applied Statistics
DOI: 10.1214/25-aoas2036
Language: English
Subfield: Statistics and Probability
Field: Mathematics
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W4417045943