Zero-Shot Extraction of Structured Clinical Data from Free-Text CT Reports Using Foundational Large Language Models
Year: 2026
Type: article
Abstract: Introduction: Large language models (LLMs) are being explored for multiple applications in medical research, including medical text classification. Authors of the study evaluated the performance of three off-the-shelf LLMs in classifying unstructured free-text CT pulmonary angiography reports for pulmonary embolism (PE)-related diagnostic labels. Methods: Authors assessed human-labeled CT reports using three LLMs (ChatGPT-4o, Llama 3.3 70B, and Llama 3.1 8B) with zero-shot simple and chain-of-thought (CoT) prompts. Classification tasks included acute PE (N = 1,025) and, among reports with PE (n = 660), bilateral PE and large PE. Results: LLMs demonstrated high classification performance for acute PE and bilateral PE but lower performance for large PE. With simple prompts, the highest F1 scores were 97.0% for acute PE (Llama 3.1 8B; sensitivity, 95.6%; specificity, 97.3%), 92.9% for bilateral PE (ChatGPT-4o; sensitivity, 98.0%; specificity, 88.6%), and 72.8% for large PE (ChatGPT-4o; sensitivity, 82.9%; specificity, 82.2%). CoT prompting did not meaningfully improve acute PE classification but provided slight improvements in subclassification performance for bilateral and large PE. Conclusions: Off-the-shelf LLMs can classify free-text CT reports using simple prompt engineering and may facilitate automated data extraction for medical research following appropriate evaluation of classification performance.
Source: Kansas Journal of Medicine
Institutions: University of Kansas, University of Kansas Medical Center
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: diamond
OpenAlex ID: W7222651799