Beyond accuracy: a three-stage evaluation of large language models for postpartum health consultation and nursing decision support
Year: 2026
Type: article
Abstract: Background Large language models may support postpartum health consultation and nursing care plan development, but their performance across different task types remains uncertain. This study compared the performance of ChatGPT 5.5 and DeepSeek V3 in these settings. Methods A three-stage cross-sectional evaluation was conducted, comprising repeated querying of 30 guideline-based questions, consultations involving 120 postpartum women, and 120 de-identified retrospective clinical cases. Responses were evaluated by clinical experts using a blinded assessment process. In Stage 2, the postpartum women who submitted the questions also performed blinded evaluations from the user perspective. Paired-samples t tests or Wilcoxon signed-rank tests were used, as appropriate. Holm-Bonferroni correction was applied within outcome families defined by study stage and evaluator type, and intraclass correlation coefficients were calculated to assess agreement among clinical expert ratings. Results In Stage 1, ChatGPT 5.5 received a higher correctness score than DeepSeek V3 (4.00 ± 0.00 versus 3.88 ± 0.33; Holm-adjusted p = 0.002). DeepSeek V3 received higher scores for completeness and clinical management appropriateness (both Holm-adjusted p < 0.001) and safety (Holm-adjusted p = 0.001), although its responses were more difficult to read. In Stage 2, ChatGPT 5.5 received higher patient ratings for satisfaction and usefulness and higher clinician ratings for correctness, completeness, helpfulness, and safety, whereas DeepSeek V3 received a higher clinician-rated readability score (all Holm-adjusted p < 0.001). In Stage 3, ChatGPT 5.5 received higher scores for clinical appropriateness, comprehensiveness, actionability, safety, and triage appropriateness (all Holm-adjusted p < 0.001), whereas DeepSeek V3 received a higher individualization score (Holm-adjusted p = 0.021). Conclusion LLM performance varied depending on the task, evaluation metric, and use case. ChatGPT 5.5 received higher scores across most clinically relevant dimensions in real-world postpartum consultations and nursing care plan generation, whereas DeepSeek V3 received higher scores for completeness and safety in standardized questions, clinician-rated readability in real-world consultations, and individualization in care-plan generation. LLMs may support health education and preparation of initial nursing care plan drafts, but their outputs require clinical review.
Source: Frontiers in Public Health
Authors: Xilin Zhao, Xinwen Zhang, Lei Liu
Language: English
Subfield: Health Informatics
Field: Medicine
Domain: Health Sciences
Open Access status: gold
Estimated APC paid: $3,801
OpenAlex ID: W7223061133