Work

Extended-reasoning mode and large language model accuracy on the Orthopaedic In-Training Examination: a paired comparison of two ChatGPT configurations

Year: 2026

Type: article

Abstract: Background Orthopaedic surgery residents are increasingly use large language models (LLMs) for studying and board preparation. Extant data compare models without isolating the effects of inference-time reasoning depth within a single model on an identical item set. We aimed to determine whether extended-reasoning inference improves LLM accuracy on the Orthopaedic In-Training Examination (OITE) relative to standard-mode output from the same model, and to characterize the failure behavior relevant to residents using these tools. Methods Cross-sectional paired benchmarking study. The 203 publicly released items of the 275-item 2024 OITE, obtained through American Academy of Orthopaedic Surgeons (AAOS) ResStudy, were each administered twice to a single multimodal LLM (GPT-5; OpenAI, San Francisco, CA) through the native ChatGPT multimodal interface between November 2025 and January 2026, once in standard mode and once in extended-reasoning mode. Prompting conditions were held identical and scored against the official AAOS answer key. Paired accuracy was compared with McNemar’s exact test; subspecialty differences with the Fisher-Freeman-Halton exact test. Results Accuracy was 79% (161/203) in standard mode and 93% (189/203) in extended-reasoning mode. Every item answered correctly in standard mode was also correct in extended-reasoning mode; all 28 discordant pairs favored extended reasoning (McNemar’s exact p < 0.001; absolute improvement 13.8 percentage points, 95% CI 9.0–18.5). Subspecialty accuracy ranged from 50% to 100% (standard) and 83% to 100% (extended reasoning), without significant between-subspecialty differences (p = 0.11 and p = 0.68). Paired improvement was significant for hand (50% to 83%; p = 0.031) and trauma (78% to 92%; p = 0.016). Accuracy did not differ between image-containing and text-only items in either configuration (both p = 1.0). Answer–rationale discordance occurred in 2% of items; incorrect answers were delivered without expressed uncertainty, at times with citations. Conclusions Inference-time reasoning depth materially changes LLM accuracy on orthopaedic board-style questions and is a setting resident’s control. Confidently delivered errors persisted under extended reasoning, supporting use as a supervised adjunct rather than a primary study resource.

Source: Journal of Orthopaedic Surgery and Research

DOI: 10.1186/s13018-026-07359-1

Authors: Claire A. Donnelley, Stephanie V. Kaszuba, Peter Noback, Xuan Luo