Year: 2026
Type: article
Abstract: As there is growing interest in utilizing large language models (LLMs) in healthcare education, there are still concerns about their accuracy. The primary study question is to evaluate the accuracy of five LLMs in answering infectious disease (ID) related questions. We compared five LLMs—ChatGPT-4.1, ChatGPT-4o-mini, Claude, Copilot, and Gemini—on 227 exam items from an infectious diseases (ID) course in a Doctor of Pharmacy program. Items were categorized by topic, Bloom’s taxonomy, and format. LLM accuracy across different categories was summarized using descriptive statistics. The Chi-Square test was used to compare results among the different LLMs. Agreement across responses from different LLMs was investigated via Cohen’s Kappa Coefficient. The overall accuracy across all models ranged from 89.4% to 91.2% with ChatGPT 4o-mini scoring highest and Claude and Copilot scoring lowest. LLMs had highest accuracy answering pharmacology questions (93.6%), followed by therapeutics (93.0%), PK/PD (91.1%), microbiology (87.5%), and medicinal chemistry (84%). Medicinal chemistry and microbiology performed significantly worse compared to therapeutics (p < 0.05). Questions categorized as Evaluate/Create performed significantly worse compared to Remember/Understand category (85.6% vs. 92.0%, p < 0.05). The accuracy of questions with a picture was significantly worse than those with no picture (69% vs 91%, respectively). Inter-model agreement was moderate (κ = 0.497–0.613). Contemporary LLMs show high accuracy on ID coursework, with weaknesses in chemistry/microbiology content, higher-order reasoning, and image interpretation; careful integration and guidance are warranted.
Source: Scientific Reports
Authors: Jenana Maker, Myo Kyong Kim, Xin Guo
Institution: University of the Pacific
Language: English
Subfield: Health Informatics
Field: Medicine
Domain: Health Sciences
Open Access status: gold
Estimated APC paid: $2,690
OpenAlex ID: W7225460505