Benchmarking freely accessible large language models for pediatric epilepsy care: a prospective mixed-methods study
Year: 2026
Type: article
Abstract: Background: Large language models (LLMs) may support patient education and clinical documentation, but their accuracy, safety, and usability in pediatric epilepsy remain insufficiently characterized. This study aimed to compare four freely accessible LLMs using standardized pediatric epilepsy scenarios and to evaluate the top-performing model that met the prespecified safety criterion in a supervised clinician workflow assessment. Methods: We conducted a prospective, two-phase mixed-methods study at the University Hospital at King Abdulaziz University in Jeddah. In phase 1, four freely accessible LLMs (ChatGPT, Gemini, DeepSeek, and Perplexity) were evaluated using 10 standardized pediatric epilepsy scenarios. Outputs were assessed by pediatric neurologists across factual accuracy, comprehensiveness and relevance, safety, clinical appropriateness, readability and caregiver tailoring, and transparency and sources. Critical safety flags were recorded separately. In phase 2, the model with the highest observed composite score among systems meeting the prespecified safety criterion underwent a supervised clinician workflow assessment using the System Usability Scale (SUS), a caregiver-facing educational-material assessment, and semi-structured interviews. Results: 9.1%). Conclusions: In this specialty-specific benchmark, the evaluated LLMs showed substantial differences in observed accuracy and safety, with DeepSeek achieving the highest observed composite score and no critical safety flags. However, the single-rater-per-model design, small sample, model updates, and limited workflow assessment preclude definitive conclusions about comparative model superiority. These findings support specialty-specific evaluation of LLMs that prioritizes safety and clinical accuracy alongside usability. Larger, independently rated, multi-center studies with repeated model sampling and objective workflow and patient-centered outcomes are needed before routine clinical implementation can be considered.
Source: Translational Pediatrics
Authors: Osama Muthaffar, Anas Alyazidi
Language: English
Subfield: Health Informatics
Field: Medicine
Domain: Health Sciences
Open Access status: diamond
OpenAlex ID: W7221143739