Paradoxical inflation of prediction model performance measures in minority patient subgroups: the role of upstream bias
Year: 2026
Type: article
Abstract: Predictive models in healthcare AI are expected to perform worse in underrepresented subgroups due to upstream biases such as underrepresentation, differential care access, and potential measurement error. In a previously published model predicting persistent postoperative opioid use, we surprisingly observed substantially higher model performance in Black patients (which comprised less than 5% of the sample) than in White patients. In this study, we used subgroup matching to investigate whether differences in patient characteristics contributed to the observed disparity, and replicate our findings in a second independent model. Across both models, performance differences - measured by AUC and other performance metrics - were attenuated after matching, suggesting that upstream biases may paradoxically inflate apparent model performance in minoritized patient groups. These findings challenge conventional assumptions about bias and highlight the importance of investigating performance differences using subgroup-level covariate alignment. Our results emphasize the need for careful evaluation of subgroup model metrics to ensure fair and trustworthy deployment of clinical AI.
Source: npj Health Systems
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: gold
Estimated APC paid: $2,790
OpenAlex ID: W7223141621