Work

Performance of ChatGPT-5 under a minimal binary prompt for detection of distal radius fractures on plain radiographs

Year: 2026

Type: article

Abstract: BACKGROUND: General-purpose multimodal large language models (LLMs) are being applied to radiology, but their performance under the simplest realistic clinical query (a single binary fracture-or-not prompt) has not been rigorously quantified. We evaluated ChatGPT-5 on this task using a public distal radius dataset. METHODS: In this retrospective diagnostic accuracy study, 396 anteroposterior distal radius radiographs from the Stanford MURA dataset were submitted to ChatGPT-5, each in a fresh temporary chat session. The prompt was: "Is there a fracture in this X-ray? Answer with yes or no." The reference standard was an image-level label assigned by consensus between a senior orthopedic resident and a board-certified orthopedic surgeon; MURA study-level labels were used only to select candidate images. Twenty-six radiographs (6.6%) returned a hedged response and were excluded from the primary analysis, leaving 370; all 26 were added back in three sensitivity analyses. Outcomes were sensitivity, specificity, PPV, NPV, F1, balanced accuracy, and Cohen's kappa, with 95% Wilson confidence intervals and McNemar's test for error asymmetry. RESULTS: = 4.06, p = 0.044). Non-binary responses occurred in 7.6% of fracture-positive and 5.7% of fracture-negative radiographs (p = 0.54); counting all as errors lowered sensitivity to 47.3% (40.2-54.5), and no assumption raised it above 54.9%. Projected to 20% prevalence, PPV was 30.3% and NPV 85.2%. CONCLUSION: Under a minimal binary prompt and without task-specific fine-tuning, ChatGPT-5 missed approximately half of distal radius fractures and fell well below published performance for dedicated CNN-based fracture detectors. Results are specific to this model version, this prompt, this anatomical region, and this dataset; they constrain claims about autonomous, minimal-prompt use rather than the broader potential of engineered LLM-based radiological workflows.

Source: Emergency Radiology

DOI: 10.1007/s10140-026-02550-7

Authors: Bana Awad, Yaniv Keren, Dror Ronel

Institutions: Technion – Israel Institute of Technology, Rambam Health Care Campus

Language: English

Topic: Artificial Intelligence in Healthcare and Education

Subfield: Health Informatics

Field: Medicine

Domain: Health Sciences

SDG: Good health and well-being

Open Access status: hybrid

Estimated APC paid: $3,990