Work

Leveraging large language models to generate multiple-choice questions for epilepsy education

Year: 2026

Type: article

Abstract: Background: Large language models (LLMs) are increasingly used to generate medical assessment items, but their reliability for disease-specific question writing remains uncertain. We benchmarked seven LLMs for epilepsy single-best-answer multiple-choice question generation under a standardized blueprint, focusing on structural compliance, clinical accuracy, expert-rated quality and educational usability. Methods: Each model completed the same 60 prespecified tasks spanning foundational knowledge, clinical cases, treatment decision-making, and diagnostic evaluation and longitudinal management, yielding 420 questions. Tasks were delivered through an identical single-turn prompt without model-specific optimization, web search or external tools. Objective evaluation assessed structural compliance, keyed-answer accuracy, textual characteristics and response time. Two blinded clinical reviewers independently rated content rigor, clarity, distractor quality, educational usefulness, cognitive alignment and overall usability on five-point scales. Results: After one standardized corrective prompt, all 420 questions met structural requirements, although first-pass compliance ranged from 83.33% to 100%. Keyed-answer accuracy varied substantially, from 61.67% to 98.33%, with Claude-5 achieving the highest accuracy. GPT-5.6 obtained the highest composite expert score (median, 3.75), followed by Claude-5 (3.67) and Kimi-K3 (3.50). Distractor quality was a common limitation, with median scores of only 2-3 across models. Only 2 of 420 questions (0.48%) were judged directly usable without revision, both generated by GPT-5.6; 119 (28.33%) were usable after minor revision and 299 (71.19%) required substantial revision. Longer outputs were associated with better cognitive alignment but not with content rigor or clarity, while response time was unrelated to overall usability. Conclusion: Contemporary LLMs can reliably reproduce the structure of epilepsy MCQs, but structural compliance and answer accuracy do not ensure educational readiness. Model strengths were multidimensional, and weak distractor construction remained a major barrier to direct use. LLMs may support expert-supervised item drafting, but independent clinical review and subsequent learner-based psychometric validation remain necessary before deployment in medical education.

Source: Frontiers in Medicine

DOI: 10.3389/fmed.2026.1978708