Safety and quality of patient-facing frozen shoulder information from four generative AI chatbot interfaces: a cross-sectional study
Year: 2026
Type: preprint
Abstract: Patients increasingly use generative artificial intelligence (AI) chatbots for musculoskeletal health information, but their safety and suitability for frozen shoulder education are uncertain. We submitted 50 prespecified patient- and caregiver-facing questions once to each of ChatGPT, Gemini, DeepSeek, and Doubao on 17 July 2026, yielding 200 first responses. Five clinicians independently assessed deidentified responses against question-specific evidence standards for safety, accuracy, empathy, information quality, transparency, and global quality; readability was calculated computationally. Twenty-one responses (10.5%) were classified as potentially unsafe. Interface-specific proportions ranged from 6.0% to 14.0%, without a significant overall difference (P = 0.442). Accuracy differed across interfaces (P < 0.001; Kendall's W = 0.639), as did empathy (P < 0.001; W = 0.580). ChatGPT had the highest median accuracy (4.40), DISCERN (60.00), and JAMA benchmark (3.00) scores, whereas Gemini had the highest median binary-adapted EQIP score (83.00) and the most favorable readability profile. All interfaces exceeded commonly recommended reading levels. No interface was uniformly superior: each produced potentially unsafe content and limited response-level transparency. These tools may support patient education but should not replace individualized clinical assessment and guidance.
Source: Research Square
Institution: Nanjing Drum Tower Hospital
Language: English
Subfield: Health Informatics
Field: Medicine
Domain: Health Sciences
Open Access status: green
OpenAlex ID: W7225036936