Failure detection in biomedical image classification under realistic distribution shifts: insights from a large-scale evaluation
Year: 2026
Type: preprint
Abstract: ABSTRACT Biomedical images (BMI) exhibit variability due to different acquisition protocols, devices, and patient populations, making failure detection at inference time essential for reliable deployment of clinical classifiers. As existing evaluations of failure detection methods use different settings, it is difficult to compare results and identify the best strategy, if any. We present a comprehensive evaluation of eight confidence scoring functions and two score-aggregation strategies across eight BMI tasks spanning diverse modalities, backbone architectures, training setups, and failure sources. The confidence ranking ability and classification error mitigation are jointly evaluated. While no single method systematically dominated across settings, aggregation of confidence scores consistently matched or approached the best individual method and substantially reduced silent failure rate. The failure detection performance was strongly correlated with classifier accuracy for all tested settings. These findings provide large-scale evidence of the strengths and limitations of confidence scoring strategies and offer actionable guidance for mitigating silent failures under realistic distribution shifts in BMI. Graphical Abstract Highlights Failure detection performance is independent of biomedical image modality but strongly correlated with classifier accuracy. Monte Carlo Dropout does not outperform simple margin-based scores despite substantially higher computational costs. Aggregating standardized confidence scores provides more robust performance than individual methods and reduces mean silent failure rates from 24% down to 8% for realistic distribution shifts. Cross-validation ensembling improves both classification accuracy and failure detection performance.
Source: medRxiv
Language: English
Subfield: Health Informatics
Field: Medicine
Domain: Health Sciences
Open Access status: green
OpenAlex ID: W7160306460