Comparative evaluation of ChatGPT, DeepSeek, and Doubao for pediatric IgA vasculitis patient education: content quality, reliability, readability, understandability, and actionability
Year: 2026
Type: article
Abstract: Background Large language model (LLM) chatbot platforms are increasingly used for health information, but their performance for pediatric immunoglobulin A vasculitis (IgAV) patient education is unclear. Objective To compare ChatGPT, DeepSeek, and Doubao across content completeness, quality and reliability, readability, understandability, and actionability. Methods Thirty-two standardized English questions were submitted to ChatGPT, DeepSeek, and Doubao on August 9, 2026, using a new conversation for each question and retaining the first complete response (96 responses). Two reviewers independently performed the original assessment, with third-reviewer adjudication. GQS, mDISCERN, QAMAI, six readability metrics, PEMAT-P, and ICC(2,1) were used. During revision, a question-specific clinical checklist was established and applied to all archived answers. The research team completed the revision scoring and confirmed the final omission results. Results Clinically relevant information omissions occurred in 16/96 responses (16.67%): 0/32 for ChatGPT, 5/32 for DeepSeek, and 11/32 for Doubao. Retained ratings identified outdated medical advice in 2/96 responses (2.08%) and no real-time information errors or hallucinations. At least one recorded content deficiency occurred in 18/96 responses (18.75%). Original ICC(2,1) values ranged from 0.767 to 0.954. Overall differences were significant for GQS, mDISCERN, QAMAI, PEMAT-U, and PEMAT-A (all P < .01), but not for readability metrics. Bonferroni-adjusted contrasts showed lower ARI, FKGL, CLI, and SMOG scores for DeepSeek than ChatGPT. DeepSeek had the highest PEMAT-U score; ChatGPT had the highest PEMAT-A score. Conclusions Performance varied by evaluation domain, and clinically relevant omissions remained a safety concern. The omission findings reflect criteria established during revision and do not establish a validated platform safety ranking. LLM platforms may supplement patient education but should not replace professional medical review.
Source: Frontiers in Medicine
Authors: Ximing Fei, Jiao Zhang, Qiuye Li, Jianhui Yang
Institution: Guizhou University
Language: English
Subfield: Health Informatics
Field: Medicine
Domain: Health Sciences
Open Access status: gold
Estimated APC paid: $3,801
OpenAlex ID: W7223112708