Work

Calibration-informed metrics for instance-level predictive reliability in medical AI

Year: 2026

Type: article

Abstract: Conventional performance metrics in clinical decision support systems, such as accuracy or sensitivity, fail to reflect the reliability of individual predictions—an essential concern for clinicians operating in high-stakes environments. We introduce a calibration-informed framework featuring two novel metrics: the Local Predictive Value (LPV) and the Credible Predictive Value (CPV). LPV estimates the empirical reliability of a prediction by assessing the observed correctness frequency in the neighborhood of its confidence score. CPV refines this estimate using a Bayesian approach, integrating global predictive values as priors to produce a posterior distribution over correctness probabilities. LPV offers a descriptive, data-driven view of local reliability, while CPV provides a belief-adjusted estimate that mitigates overfitting to sparse local data. Applied to benchmark medical imaging datasets, these metrics yielded locally adaptive, interpretable reliability estimates. Divergences between LPV and CPV identified cases where local evidence was insufficient or misleading, highlighting how Bayesian smoothing improves stability against sparse or misleading local evidence. By combining local calibration with Bayesian inference, LPV and CPV advance the development of medical AI systems that are not only accurate but also interpretable and trustworthy at the individual case level. • Introduced two metrics to assess prediction reliability in clinical AI decision support. • Local Predictive Value estimates trust based on observed outcomes near the score. • Credible Predictive Value uses Bayesian inference to blend local and global evidence. • Metrics reveal where local evidence is misleading and priors improve robustness. • Applied the method to clinical datasets for interpretable, case-specific evaluation. • Proposed a calibration-aware framework for trustworthy prediction in medical AI.

Source: Artificial Intelligence in Medicine

DOI: 10.1016/j.artmed.2026.103366

Author: Federico Cabitza

Institutions: Fondazione Bruno Kessler, FBK Center for Digital Health & Wellbeing, University of Milano-Bicocca

Language: English

Topic: Artificial Intelligence in Healthcare and Education

Subfield: Health Informatics

Field: Medicine

Domain: Health Sciences

SDG: Good health and well-being

Open Access status: hybrid

Estimated APC paid: $3,480