Work

Comparative analysis of answers to real-time patient care questions: an AI chatbot (ChatGPT) versus physician

Year: 2026

Type: article

Abstract: Artificial intelligence (AI) chatbots including ChatGPT provide rapid, conversational answers to complex medical questions. Despite promising performance on standardized examinations and specialty-specific tasks, the accuracy and safety of ChatGPT’s responses to real-time patient care questions remain uncertain, particularly when compared directly to practicing physicians. Our study aimed to address this gap by evaluating the accuracy, relevance, comprehensiveness, and safety of ChatGPT-generated answers compared with physician-generated answers to clinical questions arising during routine patient care. In this prospective comparative, blinded, cross-sectional study, practicing physicians in the emergency department (ED), general medicine ward, and intensive care unit (ICU) provided real-time clinical questions generated during patient care. The same questions were then independently answered by resident, fellow, or attending physicians using only non-AI resources, while ChatGPT provided separate responses to the questions. Nine attending physicians, blinded to the source of answers, evaluated all responses using a 5-point Likert scale across six domains: accuracy, clinical relevance, understandability, comprehensiveness, potential for patient harm, and potential bias. Composite mean scores from three evaluators per question served as the primary analysis unit. Comparisons were performed using the Mann–Whitney U test. A total of 285 questions were collected (33.0% ED, 34.0% ward, 33.0% ICU). Across all settings, residents submitted the most questions (74.5% ED, 64.9% ward, 47.9% ICU). ChatGPT outperformed physicians across all six grading categories, demonstrating significantly greater accuracy, relevance, understandability, and comprehensiveness (all p < 0.01). ChatGPT’s responses were also evaluated by experienced attending physicians to be less likely to cause patient harm or exhibit bias ( p < 0.05). Evaluators correctly identified whether answers were from ChatGPT or physician with 99.3% accuracy. ChatGPT provided higher-quality answers than practicing physicians to real-time clinical questions generated during patient care. Although evaluators could readily distinguish AI-generated answers, ChatGPT’s superior performance to physicians of varying expertise highlights its potential role as a valuable source for clinical decision support, especially for physician trainees. Clinician oversight remains essential as integration into clinical workflows evolves.

Source: BMC Medical Informatics and Decision Making

DOI: 10.1186/s12911-026-03817-2