Work

Out-of-Distribution Generalization of Pretrained Multi-Label Chest X-Ray Classifiers: A Retrospective External Validation of Seven Models

Year: 2026
Type: article
Abstract: Background/Objectives: Chest radiograph classifiers are usually benchmarked within their training distribution, whereas deployment exposes them to new populations, equipment, and labeling conventions. This study quantified out-of-distribution (OOD) performance of seven public pretrained classifiers. Methods: Six DenseNet-121 models and one ResNet-50 TorchXRayVision model were evaluated without fine-tuning on 18,424 Brazilian BRAX studies for seven common labels. The in-distribution (ID) performances of the three models trained on ChestX-ray14 were measured on held-out patients. Areas under the receiver operating characteristic curve (AUROC) were compared with a paired patient-level bootstrap; calibration, operating points, and subgroups were assessed. Results: OOD macro AUROC was 0.733–0.741 for three models trained with PadChest and 0.652–0.677 for four single-source models (all between-group differences significant). Relative to held-out ID performance, macro AUROC decreased by 6.0–11.4 percentage points. Expected calibration error of default outputs (0.028–0.392) largely reflected score normalization and fell to 0.001–0.003 after recalibration; sensitivity at default operating points was 0.035–0.797. Macro AUROC was lower for anteroposterior than posteroanterior radiographs (0.499–0.605 vs. 0.691–0.796), also within age groups; residual confounding remains possible. Conclusions: In this single Brazilian cohort, ID benchmarks overestimated external performance, and default outputs and operating points did not transfer. Local validation, recalibration, and subgroup-stratified reporting are needed before clinical use.
Source: Diagnostics
Language: English
Field: Medicine
Open Access status: gold
Estimated APC paid: $3,137
OpenAlex ID: W7236487349
FWCI:
Cites:
Cited by:
Related to: 10