Work

Clinical accuracy, information quality and readability of chatbot responses to patient questions about lower extremity amputation

Year: 2026

Type: article

Abstract: ObjectiveTo compare the clinical accuracy, information quality and readability of chatbot responses to common patient questions about lower extremity amputation.MethodsTwenty frequently asked patient questions were submitted once to each of ChatGPT, Gemini and DeepSeek in separate sessions. Two reviewers, blinded to chatbot identity, independently assessed responses using a four-category clinical accuracy scale and the DISCERN instrument. Readability was quantified as the mean grade level across the Flesch-Kincaid Grade Level, Gunning Fog, Coleman-Liau, and Simple Measure of Gobbledygook indices. Comparisons used Friedman tests and Holm-adjusted Wilcoxon signed-rank tests.ResultsSixty responses were analysed. Mean (±standard deviation) clinical accuracy scores were 2.20 ± 0.57, 1.55 ± 0.65 and 1.20 ± 0.38 for ChatGPT, Gemini and DeepSeek, respectively (lower scores indicate greater accuracy). Gemini and DeepSeek received lower scores than ChatGPT (adjusted p=0.012 and p=0.002, respectively), whereas the Gemini-DeepSeek difference was not significant (p=0.090). Mean DISCERN scores were 50.05 ± 5.78, 53.33 ± 5.43 and 58.23 ± 3.39, respectively, with significant differences in all pairwise comparisons. Mean reading grades were 12.93 ± 1.28, 13.17 ± 1.36 and 11.74 ± 1.15, respectively; DeepSeek had lower grades than both comparators (both adjusted p<0.001).ConclusionDeepSeek responses received higher information quality scores and had lower reading grade levels, although clinical accuracy did not differ significantly between DeepSeek and Gemini. All three chatbots exceeded commonly recommended reading levels for patient education. Clinician review and language adaptation are warranted before chatbot-generated information is used to supplement individualised counselling on amputation and rehabilitation.

Source: Journal of orthopaedic surgery

DOI: 10.1177/10225536261504909

Authors: Kamil Balaban, Mehmet Batu Ertan, Mahmut Kalem

Institution: Ankara University

Language: English

Topic: Artificial Intelligence in Healthcare and Education

Subfield: Health Informatics

Field: Medicine

Domain: Health Sciences

SDG: Good health and well-being

Open Access status: gold

Estimated APC paid: $1,600