Work

Battle of the (Chat)Bots: Comparative Evaluation of April 2025 AI Model Accuracy on Pharmacotherapeutic Cases

Year: 2025

Type: article

Abstract: OBJECTIVE: Generative artificial intelligence (AI) tools are increasingly utilized in health care education, offering potential learning benefits amid concerns about accuracy, reasoning, and ethical use. There is limited literature within pharmacy education, particularly quantitative studies, that explores their clinical application. This study sought to evaluate and compare the performance of 3 AI tools: ChatGPT (OpenAI), Copilot (Microsoft), and Claude AI (Anthropic), in responding to clinical patient minicases to assess their ability to apply therapeutic principles and clinical reasoning. METHODS: Six patient minicases of varying complexities were selected from 2 integrated pharmacotherapeutic courses. Each AI model received identical prompts requiring therapeutic categorization (optimal, suboptimal, harmful) and rationale, with responses graded using existing course rubrics. Statistical analyses were conducted to compare performance across chatbots and rubric domains. RESULTS: Claude AI achieved the highest average (94%, SD 9%) compared to ChatGPT (72%, SD 9.4%) and Copilot (53%, SD 18.3%), with a statistically significant difference in performance in 1 pharmacotherapeutics course. Claude AI consistently outperformed across rubric domains, whereas ChatGPT showed moderate accuracy but variable reasoning depth. Copilot had the lowest performance and highest variability. Reference quality, therapeutic plan accuracy, and defense of the therapeutic plan varied among platforms. CONCLUSION: Claude AI demonstrated the strongest overall performance. However, all tools exhibited limitations, reinforcing the need for clinical expert oversight. Though AI chatbots may support pharmacy education, our findings suggest that their current ability to perform multistep analysis remains limited. They often lack the clinical judgment, experience, and context needed for nuanced decision-making.

Source: American Journal of Pharmaceutical Education

DOI: 10.1016/j.ajpe.2025.101922

Authors: Bao Anh C. Tran, Gwendolyn A. Wantuch, Moses Mangasar, Aimon C. Miranda

Institution: University of South Florida

Language: English

Topic: Artificial Intelligence in Healthcare and Education

Subfield: Health Informatics

Field: Medicine

Domain: Health Sciences

SDGs: Good health and well-being, Quality education

Open Access status: gold

Estimated APC paid: $2,400