Work

Quality, reliability, and readability of large language model responses to questions about immune checkpoint inhibitor-associated myocarditis: A cross-sectional study

Year: 2026
Type: preprint
Abstract: Immune checkpoint inhibitor-associated myocarditis (ICI-M) is a rare but often fatal immune-related adverse event. Large language models (LLMs) have provided new opportunities for patient education, yet whether they can effectively support patient education in ICI-M remains unknown. In this study, we submitted 20 standardized ICI-M-related patient questions to five LLMs (ChatGPT-5.5, Gemini-3.5 Flash, DeepSeek-V4, Qwen 3.7, and Doubao), generating 100 responses. Model performance was evaluated using the Global Quality Score (GQS), modified DISCERN (mDISCERN), Quality Analysis of Medical Artificial Intelligence (QAMAI), Ensuring Quality Information for Patients (EQIP), Patient Education Materials Assessment Tool understandability and actionability scores (PEMAT-U and PEMAT-A), and six readability indices. Significant inter-model differences were observed across most quality and readability metrics, and no single model outperformed the others across all dimensions. DeepSeek-V4 achieved the highest mDISCERN score [5.00 (IQR, 4.00–5.00)], whereas ChatGPT-5.5 achieved the highest PEMAT-A score [40.00 (IQR, 35.00–45.00)]. Readability was suboptimal, with a mean FKGL of 16.60 ± 2.64. Correlation analyses demonstrated information quality, usability, and readability represented correlated but distinct evaluation dimensions. While LLMs may serve as adjunctive tools for ICI-M patient education, substantial optimization of content accessibility and actionability is required prior to clinical implementation.
Language: English
Field: Medicine
Open Access status: green
OpenAlex ID: W7236039258
Cites: 34
Cited by:
Related to: 10