Limited transportability of a deep learning grading model for severe lumbar neural foraminal stenosis: external validation in two public cohorts
Year: 2026
Type: article
Abstract: Background Aggregate accuracy and discrimination can remain acceptable even when an artificial intelligence system misses severe disease at its applied decision rule. We tested whether a locked lumbar magnetic resonance imaging (MRI) pipeline retained severe neural foraminal stenosis sensitivity and its development-defined safety operating point in an independent public cohort. Methods A two-stage ConvNeXt-Tiny pipeline was developed using LumbarDISC sagittal T2-weighted examinations. It localized eight level-by-side targets from L2–L3 to L5–S1 and graded each as normal/mild, moderate, or severe. Target-specific temperature scaling was applied. Using five-fold out-of-fold predictions, we selected the highest severe-probability threshold that achieved the investigator-selected sensitivity target of 0.90. All components were frozen before one-time evaluation in a 395-patient internal test set and the independent 465-patient AISSLab cohort. The primary endpoint was target-level severe-class sensitivity in AISSLab. Confidence intervals used 2,000 patient-cluster bootstrap samples. Results AISSLab contributed 2,190 labeled targets, including 205 severe events. Argmax severe-class sensitivity decreased from 0.404 (61/151; 95% CI, 0.315–0.487) internally to 0.185 (38/205; 95% CI, 0.135–0.242). The external-minus-internal difference was −0.219 (95% CI, −0.314 to −0.114). External severe-class area under the receiver operating characteristic curve was 0.885, indicating that preserved ranking did not ensure sensitivity at the frozen decision rule. The safety threshold achieved 0.824 sensitivity (169/205), below its 0.90 development target, with a positive predictive value of 0.288 and referral rate of 0.268. Dataset-provided boxes increased argmax sensitivity by only 0.029. Target-domain temperature scaling slightly improved calibration but did not change argmax sensitivity and increased referral workload at the fixed threshold. Conclusion This pipeline showed limited internal severe-class sensitivity and a further decrease externally. Independent validation should assess intended operating points while accounting for reference-standard differences and incomplete labels.
Source: Frontiers in Medicine
Institutions: Hebei North University, Tianjin University of Sport
Language: English
Topic: Medical Imaging and Analysis
Subfield: Biomedical Engineering
Field: Engineering
Domain: Physical Sciences
Open Access status: gold
Estimated APC paid: $3,801
OpenAlex ID: W7235729783
