AI large language models versus physicians in psycho-oncology cancer communication: A blinded multicenter comparison
Year: 2026
Type: article
Abstract: Objectives To compare the accuracy, empathy, and helpfulness of consumer AI chatbot responses with physician-written responses to psychosocial questions raised by people with cancer; to test whether expert raters could distinguish physicians from AI responses; and to establish the Cancer Psychosocial Response Benchmark (CPRB-15) for longitudinal model evaluation. Methods Fifteen questions spanning 5 psychosocial domains were answered by ChatGPT, Claude, Gemini, DeepSeek, and 1 of 2 psycho-oncology psychiatrists. Eight psychiatry and psycho-oncology clinicians rated randomized, source-blinded responses from 1 to 5 for accuracy, empathy, and helpfulness and attempted to identify the physician response. Primary crossed linear mixed-effects models included ChatGPT, Claude, Gemini, and physician responses; DeepSeek was descriptive because it was under-sampled. Results The study yielded 420 response-level evaluations and 1,260 metric-level ratings. Compared with physician responses, adjusted composite ratings were higher for ChatGPT (+0.76; 95% CI, +0.56 to +0.96), Claude (+0.72; +0.52 to +0.92), and Gemini (+0.32; +0.12 to +0.52). Experts identified the physician response in 23 of 105 judgments (21.9%; exact 95% CI, 14.4–31.0%), which did not differ from the prespecified 20% chance rate ( p = 0.626). Higher readability grade level correlated with higher composite rating ( r = 0.61; p < 0.001). Significance of results Current chatbots can produce concise psychosocial cancer responses that expert clinicians rate highly, but these text-only findings do not establish clinical equivalence or support autonomous use. The frozen questions, responses, ratings, and blinding judgments form CPRB-15, a reusable benchmark for tracking communication quality as models evolve.
Source: Palliative & Supportive Care
Authors: Adriell Louis(Conceptualization, Methodology, Writing – original draft), Jack D. Nothnagel(Data curation, Investigation, Project administration, Writing – original draft, Writing – review & editing), Kai Parker-Fong(Formal analysis, Investigation, Methodology, Writing – original draft, Writing – review & editing), Jackson Mitchell, William Breitbart, Adrienne D. Mishkin(Investigation, Validation, Writing – review & editing), Jasmine Miller, Jon A. Levenson(Investigation, Validation, Writing – review & editing), Michael L. Yee(Investigation, Validation, Writing – review & editing), Jonathan E. Alpert(Investigation, Validation, Writing – review & editing), Yesne Alici(Investigation, Validation, Writing – review & editing), Andrew T. Drysdale(Investigation, Methodology, Supervision, Validation, Writing – review & editing), Darnell K. Adrian Williams(Conceptualization, Data curation, Formal analysis, Investigation, Methodology, Project administration, Supervision, Writing – original draft, Writing – review & editing)
Language: English
Subfield: Health Informatics
Field: Medicine
Domain: Health Sciences
Open Access status: hybrid
Estimated APC paid: $3,550
OpenAlex ID: W7236801715
