Deriving the OTA/AO fracture classification from routinely collected radiology reports using a large language model
Year: 2026
Type: article
Abstract: Objectives: Fracture classification plays a pivotal role in research and quality assurance; despite its wide acceptance, the OTA/AO classification is seldom documented in patients' electronic medical records, which impedes fracture registry creation and effective interdisciplinary communication. In this study, we investigate "off-the-shelf" large language models (LLMs) in translating free text in radiology reports into OTA/AO classification labels. Methods: We employed a Health Insurance Portability and Accountability Act-compliant LLM to classify 109 fracture descriptions from randomly selected radiology reports in a deidentified electronic medical record database. Ground-truth classifications were assigned by expert orthopaedic traumatologists based on corresponding radiographs. Multiple prompting strategies were tested, including zero-shot prompting, zero-shot chain-of-thought prompting, and retrieval-augmented generation. We additionally asked the LLM to assign classification labels to "ideal" fracture descriptions written according to the 2018 OTA/AO Fracture and Dislocation Classification Compendium. Model performance was assessed using Cohen kappa and accuracy against ground-truth labels. Results: levels. Performance declined to slight agreement at the subgroup level. The best performance was observed using ideal fracture descriptions with retrieval-augmented generation, in which the agreement between the full LLM-generated and ground-truth labels remained moderate. Classification errors were largely due to imprecise descriptions, hallucinated information, or incorrect application of factually correct information. Conclusions: Our study demonstrates some potential for LLMs to translate free-text fracture descriptions into OTA/AO classifications, allowing for efficient labeling of large datasets of radiology reports. Future work should focus on refining model classification capabilities using more sophisticated prompting methods. Level of Evidence: Level III.
Institutions: University of California, San Francisco, University of California Davis Medical Center, University of California, Davis, University of British Columbia, Loyola University Medical Center, St. Luke's University Health Network, HealthPartners, University of Missouri Health System, University of California, Los Angeles, Harbor–UCLA Medical Center
Language: English
Subfield: Health Informatics
Field: Medicine
Domain: Health Sciences
Open Access status: gold
Estimated APC paid: $1,900
OpenAlex ID: W7159974813
FWCI:
Cites:
Cited by:
Related to: