Work

Comparison of the performance of large language models (ChatGPT-5.5, Gemini 3.1, and Claude 4.6) in answering pharmacology questions

Year: 2026

Type: article

Abstract: Large language models (LLMs) have introduced a fundamental transformation in healthcare and medical education. This study aimed to evaluate the performance of LLMs in answering pharmacology questions. This quantitative cross-sectional study was conducted in Iran in 2026. A 50-item Persian multiple-choice questionnaire assessing pharmacology knowledge was administered to ChatGPT-5.5, Gemini 3.1, and Claude 4.6. Data were analyzed using confusion matrix methods, Cochran’s Q, McNemar, and point-biserial tests. Sensitivity values were 1.00 for ChatGPT-5.5, 0.94 for Gemini 3.1, and 0.90 for Claude 4.6. Specificity values were 0.43, 0.50, and 0.35, respectively. Accuracy was 0.82 for both ChatGPT-5.5 and Gemini 3.1, and 0.72 for Claude 4.6. Overall, the mean sensitivity, specificity, and accuracy across all LLMs were 0.95, 0.43, and 0.78, respectively. The analysis further revealed a statistically significant decline in performance of ChatGPT-5.5 and Gemini 3.1 when addressing quantitative questions (φ = −0.286, p < .05). LLMs have potential to support professionals in educational/knowledge contexts of pharmacology, provided ethical considerations are addressed, though further research is needed to better understand their performance and limitations.

Source: Scientific Reports

DOI: 10.1038/s41598-026-75099-7

Authors: Mohsen Khosravi, Kobra Naseri, Zahra Kiani, Fatemeh Khosravi, Karim Naraki

Institutions: Birjand University of Medical Sciences, Guilan University of Medical Sciences

Language: English

Topic: Artificial Intelligence in Healthcare and Education

Subfield: Health Informatics

Field: Medicine

Domain: Health Sciences

SDG: Good health and well-being

Open Access status: gold

Estimated APC paid: $2,690