Beyond Accuracy: A Multidimensional Evaluation of Large Language Models for Public-Facing Nutrition Questions in Hepatocellular Carcinoma
Year: 2026
Type: preprint
Abstract: Background Large language models (LLMs) are readily accessible sources of nutrition information for patients with hepatocellular carcinoma (HCC) and their caregivers. In this setting, however, an answer that is factually plausible may still be clinically incomplete, insufficiently qualified, difficult to read, or unsafe because nutrition decisions are shaped by cirrhosis, sarcopenia, ascites, hepatic encephalopathy, treatment-related adverse effects, comorbidity, and potential food-drug interactions. Objective To compare five publicly accessible chatbots across safety, accuracy, empathy, information quality, and readability and to assess the short-term numerical stability of their outputs through repeated live querying. Methods This comparative observational study evaluated ChatGPT, Gemini, Microsoft Copilot, Doubao, and DeepSeek using 62 standardized English-language questions. The question set was developed through a structured process incorporating HCC nutrition guidelines, clinical recommendations, literature retrieval, online health information resources, and analysis of frequently discussed nutrition concerns from publicly available patient-oriented forums and internet communities. Each question was submitted to all models using standardized prompts. Five independent evaluators assessed response quality using predefined criteria and validated instruments. The primary comparative analysis included 310 first-round responses, and a second live-query round using the same protocol was conducted to assess short-term numerical stability. Results The primary analysis included 310 first-round chatbot-generated responses, with high inter-rater agreement; an additional 310 repeated-query responses were used for the short-term stability analysis. Most first-round responses were classified as safe, but 47 contained potentially harmful, misleading, or clinically inappropriate content. The overall between-model safety comparison was not statistically significant, whereas accuracy, empathy, information quality, and readability differed significantly across chatbots. Safety concerns most often reflected incomplete escalation advice, overgeneralized dietary recommendations, or inadequate attention to clinical context. Repeated querying showed generally high short-term numerical stability under the study conditions. Conclusions Publicly accessible LLMs can provide useful general nutrition information in HCC, but no single performance dimension adequately captures patient-facing quality. Factual accuracy should not be interpreted as a proxy for safety, communication quality, or readability. These systems are best viewed as adjuncts to, rather than substitutes for, individualized clinical guidance; further development should prioritize context-sensitive safety warnings, source transparency, and patient-appropriate communication.
Source: Research Square
Institutions: Wuxi People's Hospital, Nanjing Medical University
Language: English
Subfield: Health Informatics
Field: Medicine
Domain: Health Sciences
Open Access status: green
OpenAlex ID: W7225821319