Work

Accuracy and explanation quality of three large language models on European anaesthesiology board examination practice questions

Year: 2026
Type: article
Abstract: Trainees and clinicians consult large language models (LLMs) informally for examination preparation, often alongside commercial practice-question texts. LLMs have not been evaluated on preparation material for the European Diploma in Anaesthesiology and Intensive Care (EDAIC), and no anaesthesiology board-examination study has combined a multi-vendor comparison with blinded, multi-rater explanation-quality assessment linked to accuracy. This study assessed both, and the reliability of the book’s reference answers. Claude Sonnet 5, GPT-5.6 Terra and Gemini 3.5 Flash were queried once on 262 questions (1310 True/False items) from a commercial EDAIC Part 1 preparation text and scored against the book’s key. Accuracy analyses accounted for clustering (generalised estimating equations, mixed-effects models, cluster bootstrap); equivalence was tested by two one-sided tests. Three blinded anaesthesiology specialists rated 495 explanations (165 items) on five axes by 2-of-3 consensus; axes with Gwet’s AC1 below 0.60 were not tested. Items on which all three models, or any model on a rated item, disagreed with the key, plus 20 random controls, were adjudicated blind against primary guidelines. Accuracy was 89.9% to 91.3%; all paired differences and confidence intervals lay within 3% points, meeting the pre-specified 5-point equivalence margin ( p < 0.001). Adjudication judged the book’s key incorrect or contested for 59 of 81 such items, against 1 of 20 controls; excluding them raised accuracy to 93.8–95.5% without altering the equivalence. Completeness (AC1 0.428) was not tested; on the three tested axes no between-model difference survived multiplicity correction ( p ≥ 0.305); confidence intervals reached 11% points, and two excluded zero before correction. Incorrect answers were associated with poorer explanations in pooled tests not adjusted for item clustering, but discrimination was poor: negative predictive value 89.3% to 90.9% against an 88.5% base rate, and area under the receiver operating characteristic curve 0.622 (95% CI 0.533–0.724). Three mainstream-tier LLMs were equivalent in accuracy on EDAIC Part 1 practice items, although the book’s key was often unsound where they disagreed; on explanation quality no difference survived correction, and equivalence was not established. Explanation quality identified incorrect answers too unreliably to serve as a check; informal use should retain verification against authoritative sources.
Language: English
Field: Medicine
Open Access status: gold
Estimated APC paid: $2,990
OpenAlex ID: W7237350813
FWCI:
Cites:
Cited by:
Related to: