Work

A SNAPpy use of large language models: using large language models to classify treatment plans in pediatric acute otitis media

Year: 2025

Type: article

Abstract: BACKGROUND AND SIGNIFICANCE: Acute otitis media (AOM) is a leading cause of pediatric antibiotic overuse. Safety Net Antibiotic Prescriptions (SNAPs) are recommended for antibiotic stewardship but are difficult to identify due to lack of structured documentation. OBJECTIVE: This study validates the accuracy of Versa, a GPT-4o based HIPAA-compliant large language model (LLM), to classify AOM treatment plans from physician notes. METHODS: A retrospective cross-sectional study analyzed pediatric AOM encounters. Multiple prompting strategies were used to classify treatment plans and validated against a representative sample of manual reviews by 2 pediatricians. A locally fine-tuned model, Clinical-Longformer was also trained and tested against Versa and human review. RESULTS: In total, 5707 encounters were included; 374 reviewed manually. Zero-shot accuracy was 97.8%; few-shot accuracy was 85%. Clinical-Longformer achieved 93.3% accuracy. CONCLUSION: Versa effectively identifies AOM treatment plans, providing a cost-efficient quality improvement tracking tool for prescription practice patterns in pediatric antibiotic stewardship efforts.

Source: Journal of the American Medical Informatics Association

DOI: 10.1093/jamia/ocaf170

Authors: Jessica J Pourian, Ben Michaels, Anh Vo, A Jay Holmgren, Augusto Garcia-Agundez, Valerie Flaherman

Institution: University of California, San Francisco

Language: English

Topic: Machine Learning in Healthcare

Subfield: Artificial Intelligence

Field: Computer Science

Domain: Physical Sciences

SDG: Good health and well-being

Open Access status: hybrid

Estimated APC paid: $4,332