Coin-flip deference and a small reliability response: a partial replication in medical vision–language models
Year: 2026
Type: preprint
Abstract: Background. Earlier work reported that medical vision–language models scarcely adjust their use of diagnostic advice to its stated reliability. We tested the decision-level consequences with a separate pipeline, new patients and generated answers. Methods. A frozen local protocol, not a preregistration, specified 400 NIH ChestX-ray14 patients, balanced for effusion and selected after excluding patients with ChestX-Det thumbnail matches. MedGemma-4B and Lingshu-7B answered with no advice or either binary report from a simulated tool at 3 stated symmetric reliabilities. The primary arm disclosed the simulated error mechanism; a metadata-only arm omitted it. Primary endpoints were the change in counterfactual decision reliance, ∆R = R(0.99) − R(0.50), and the loss of balanced correct decisions with coin-flip advice, H50. We used class-stratified patient bootstrap intervals. Results. In the primary arm, coin-flip reliance was 19.8 [16.3, 23.3]% for MedGemma and 34.5 [30.8, 38.0]% for Lingshu (95% intervals). The corresponding H50 values were 6.4 [4.4, 8.4] and 10.0 [7.4, 12.6] pp. Both fell below always following a 99% tool, by 13.4 and 8.1 pp. MedGemma's ∆R was -0.3 [-1.5, 0.8] pp; Lingshu's was positive, 3.7 [2.0, 5.7] pp. Both 90% intervals lay inside the prespecified ±10 pp margin, but Lingshu's detectable response challenges blanket reliability blindness. Its positive response was supported only in the mechanism-explicit arm. Conclusion. Deference to disclosed uninformative advice and under-use of highly reliable advice replicated. Blanket blindness did not replicate for Lingshu. Generated decisions establish neither probability-level blindness nor clinical readiness; NLP-mined labels, selected normal negatives and possible training exposure limit interpretation.
Author: Mamun, S M Abdulla Al
Institution: University of Manchester
Language: English
Subfield: Health Informatics
Field: Medicine
Domain: Health Sciences
Open Access status: green
OpenAlex ID: W7236736676
