Multidimensional evaluation of large language models for patient education on asthma during pregnancy: a cross-sectional comparative study
Year: 2026
Type: article
Abstract: Introduction Patients increasingly use large language models (LLMs) for health information, but their suitability for high-risk reproductive medicine remains uncertain. We compared six LLMs on patient-facing questions about asthma during pregnancy across clinician-rated overall clinical response quality, information quality, understandability, actionability, readability, and safety. Methods In this cross-sectional comparative study, a two-stage modified Delphi-informed process reduced an initial 40-question pool to 30 candidates and 19 retained questions. Six LLMs answered each question once under a standardized structured prompt, yielding 114 responses and 342 rater-level records. Three clinical evaluators assessed clinician-rated overall clinical response quality, DISCERN, and PEMAT understandability and actionability; four readability indices were calculated. Models were compared using matched question-level nonparametric tests with multiplicity correction and effect sizes. In a revision-stage safety analysis, two clinicians independently rated all 114 responses using a structured safety taxonomy; responses with rating disagreements or prespecified severity triggers underwent blinded adjudication by a third senior clinician. Results Clinician-rated overall clinical response quality differed across models [Friedman χ 2 (5, N = 19) = 72.31, P < 0.001; Kendall's W = 0.76]. GPT-5.5 and Claude Sonnet 5 had the highest observed median scores, without a significant difference after correction. DISCERN, PEMAT, and readability also differed across models, but no model led consistently across outcomes. None achieved a median Flesch-Kincaid grade level at or below grade 8. Inter-rater agreement for the presence of any safety concern was 87.7% (Cohen's kappa=0.70). After adjudication by a third senior clinician, 37/114 responses (32.5%) had at least one safety concern, including 4/114 (3.5%) rated as high-risk unsafe advice. Conclusions Under structured prompting, LLM performance was promising but uneven across selected asthma-in-pregnancy questions. High clinician-rated overall clinical response quality did not ensure superior information quality, usability, readability, or safety. The results do not establish safety for unsupervised patient use. LLM-generated patient information should undergo guideline-based clinical verification, explicit safety review, and readability editing before deployment.
Source: Frontiers in Digital Health
Authors: Zhoujun Zhu, Cong Xie, Haiyan Wu
Language: English
Subfield: Health Informatics
Field: Medicine
Domain: Health Sciences
Open Access status: gold
Estimated APC paid: $3,077
OpenAlex ID: W7225831921