Performance of LLM-based systems in a national dental specialty examination: a multi-system benchmark study
Year: 2026
Type: article
Abstract: Large language models (LLMs) are increasingly evaluated in healthcare and medical education, particularly for examination-based tasks. However, their performance in dental specialty examinations and across languages remains uncertain. This study compared response accuracy across LLM-based systems using Turkish and researcher-translated English versions of a retrospectively assembled DUS-sourced question set in Türkiye. A retrospective DUS-sourced item set was assembled across nine examination administrations (2020–2025). Question texts were obtained from archived booklets and commercial preparatory compilations; final answers and cancellation status were verified against ÖSYM materials. After exclusion of 17 image-dependent and 3 officially cancelled questions, 1,060 source questions were retained. Six LLM-based systems were evaluated: ChatGPT, ChatGPT Pro (Pro-tier access condition), ChatGPT Dentistry (third-party custom GPT), DeepSeek, Microsoft Copilot, and Google Gemini. Each system was evaluated in both Turkish and English, yielding 12,720 system–question–language responses. Binary response accuracy was analysed using generalised estimating equations with source-question clustering to account for repeated evaluations of the same questions. System × language, system × discipline, and system × question-format interactions were evaluated, together with the overall effect of examination period; Holm adjustment was applied to relevant pairwise comparisons. Overall accuracy ranged from 31.5% for Copilot to 71.7% for ChatGPT Dentistry. The system × language interaction was significant (Wald χ²(5) = 158.73, p < 0.001), indicating system-dependent language effects. Accuracy was lower in Clinical Sciences than in Basic Sciences across all systems, with a significant system × discipline interaction (Wald χ²(5) = 101.06, p < 0.001). Combination-type multiple-choice questions were associated with lower odds of a correct response than standard multiple-choice questions (odds ratio = 0.79, 95% confidence interval 0.67–0.94, p = 0.008), whereas the system × question-format interaction was not significant ( p = 0.307). Accuracy also varied across examination periods ( p < 0.001) without a consistent temporal trend. This study provides an examination-specific benchmark of LLM answer-selection accuracy in DUS, with performance varying across systems, languages, disciplinary domains, and question formats. It does not provide evidence that LLM use improves learning outcomes or establishes clinical competence or patient safety.
Source: BMC Oral Health
Institution: Usak University
Language: English
Subfield: Health Informatics
Field: Medicine
Domain: Health Sciences
Open Access status: gold
Estimated APC paid: $3,190
OpenAlex ID: W7221763034
FWCI:
Cites:
Cited by:
Related to: