Benchmarking Large Language Models on the Turkish Dermatology Board Exam: A Comparative Multilingual Analysis
Year: 2025
Type: article
Abstract: Artificial intelligence (AI) technologies are used as an effective tool in medical education to support the acquisition of theoretical knowledge and improve the clinical skills of medical students and residents.1 The increasing use of AI applications in medical education has potential, especially in disciplines based on visual diagnoses, such as dermatology; this trend necessitates re-evaluating conventional assessment tools, such as specialty competency exams, with AI models. 2 Studies on the performance of AI models in medical education exams reveal the potential, limitations, and room for improvement of this technology.3 Recent benchmark studies show that state-of-the-art large language models (LLMs) (e.g., GPT-4, Gemini Advanced, Claude) can exceed the 60% pass mark on united states medical licensing examination step 1-style items and perform at or near the resident level in ophthalmology and orthopedic vignette sets.[3][4][5] Despite the progress made, two critical knowledge gaps remain.First, almost all validation studies have been conducted in English, while more than half of the world's medical AbstractAim: Large language models (LLMs) are increasingly integrated into medical education; however, their performance on dermatology examinations in non-English contexts has not been extensively studied.This study aimed to evaluate the performance of six LLMs in terms of accuracy, error profile, and response time on the Turkish Dermatological Society (TDS) qualifying examination. Materials and Methods:Two hundred publicly available multiple-choice questions from the TDS exam were submitted to six LLMs (ChatGPT-4, Gemini-2.0,Claude-3.7,Grok-3, DeepSeek-R1, Qwen-2.5).Each model was tested in Turkish and in English, under both batch and single-item prompt formats.The strengths and weaknesses of the models were tested under different conditions.Results: Claude-3.7 and Grok-3 performed best (~83-84% correct) with low variance, whereas Qwen-2.5 and DeepSeek-R1 had lower accuracy (~75%) with more simple errors.Across all models, switching from Turkish to English increased median accuracy by 19.5% (P = 0.028).In contrast, batch vs. single-item prompting showed no overall performance difference (P = 0.280).DeepSeek-R1 was markedly slower (≥ 10 minutes per question vs ~134 seconds for others, P < 0.001).All models achieved high accuracy on common conditions but struggled with nuanced cases and negatively phrased questions. Conclusion:Current LLMs can answer standard dermatology certification questions with moderate to high accuracy, especially in English.However, they are still susceptible to linguistic traps, negation, and nuanced clinical distinctions.Before they can be routinely used for educational or clinical purposes, optimization for Turkish language input and complex reasoning is necessary.
Source: Turkish Journal of Dermatology
Authors: Ahmet Uğur Atılan, Niyazi Çetin
Language: English
Subfield: Health Informatics
Field: Medicine
Domain: Health Sciences
Open Access status: diamond
OpenAlex ID: W4414096308