Work

Institution-specific CT protocol assignment from referral text: A comparison of prompt-engineered and fine-tuned transformer-based language models

Year: 2026

Type: article

Abstract: Purpose Radiological vetting involves reviewing imaging referrals to confirm their appropriateness and assign an acquisition protocol. Growing workforce shortages have turned this vital step into a resource-intensive bottleneck. While Large Language Models (LLMs) have shown promise in applying generic appropriateness criteria, their ability to adapt to institution-specific protocols remains unexplored. This study evaluates a generative LLM (GPT-4o-mini) and a medically pretrained RoBERTa model on institution-specific CT protocol assignment from referral text based on local clinical guidelines Methods In this retrospective study, CT requests at our institution from 2023 were categorized into 17 protocol labels, focusing on thoracic and urogenital examinations. We evaluated GPT-4o-mini and a fine-tuned RoBERTa model against the original vetting by radiologists. Primary outcomes included overall agreement and per-class performance metrics. We compared the model performance using McNemar's test and estimated the effect sizes using odds ratios. Results Among 16,926 referrals, RoBERTa achieved a higher overall agreement with the reference standard (95.7%, 736 mismatches) compared to GPT-4o-mini (95.2%, 802 mismatches). Both models performed well for frequent protocols, with renal stone and low-dose chest CTs achieving PPV ≥0.98. Macro-averaged metrics showed a larger gap between the models: balanced accuracy was 86.1% for RoBERTa versus 70.1% for GPT-4o-mini, and macro F1 was 74.7% versus 59.8%, respectively. Conclusion Both fine-tuned and prompt-based models demonstrated high adherence to the local vetting guidelines. The fine-tuned RoBERTa model performed better on most low-frequency protocols (macro F1 74.7% vs 59.8%), whereas the prompt-engineered approach offers a flexible alternative that avoids per-institution retraining.

Source: European Journal of Radiology Artificial Intelligence

DOI: 10.1016/j.ejrai.2026.100136

Authors: Jack Junchi Xu, Mathias Willadsen Brejnebøl, Helene Mygind Wolsk, Sidse Schwartz Balle, Felix Christoph Müller, Michael Brun Andersen

Institutions: Copenhagen University Hospital, Gentofte Hospital, Frederiksberg Hospital, Bispebjerg Hospital

Language: English

Topic: Artificial Intelligence in Healthcare and Education

Subfield: Health Informatics

Field: Medicine

Domain: Health Sciences

SDG: Good health and well-being

Open Access status: gold

Estimated APC paid: $2,900