Performance comparison of artificial intelligence-based language models on oral pathology questions in dental education
Year: 2025
Type: article
Abstract: Aims: This study aimed to comparatively evaluate the diagnostic accuracy levels of current large language models (LLMs)—including ChatGPT-3.5, ChatGPT-4, Gemini, and Deepseek—through multiple-choice questions related to oral pathology, a core subject in undergraduate dental education. Methods: A total of 30 multiple-choice questions covering oral pathology topics within the dental curriculum were prepared by a domain expert using current textbooks and course materials. These questions were applied in a text-based format to four LLMs (ChatGPT-3.5, ChatGPT-4, Gemini, Deepseek), and only the first responses were recorded without any prompting or correction. Diagnostic accuracy was assessed based on the percentage of correct answers, and inter-model agreement was evaluated using the Kappa statistic. A p-value < 0.05 was considered statistically significant. Results: ChatGPT-3.5 and ChatGPT-4.0 each achieved an accuracy rate of 93.33%, with corresponding kappa values of ?=0.951 and ?=0.927, respectively. Gemini reached 86.67% accuracy (?=0.879), while Deepseek achieved 80% accuracy (?=0.855). All models showed statistically significant agreement with the reference answers (p=0.000). Conclusion: This study demonstrated that ChatGPT-3.5 and ChatGPT-4.0 outperformed other models in terms of diagnostic accuracy and agreement in answering fundamental oral pathology questions. Although all models showed significant concordance, ChatGPT-based models stood out in terms of overall accuracy. However, considering potential error margins, these models should be used only as complementary tools in dental education and should not replace primary sources of knowledge acquisition for students.
DOI: 10.51271/jdse-0052
Author: Katibe Tuğçe Temur
Language: English
Subfield: Health Informatics
Field: Medicine
Domain: Health Sciences
Open Access status: hybrid
OpenAlex ID: W4411031334