Work

Patient-Facing AI Chatbots vs Primary Care Physicians: A Standardized Patient Field Experiment in China

Year: 2026

Type: preprint

Abstract: Importance Primary health care (PHC) systems worldwide face persistent workforce shortages, access barriers, and quality gaps. As patients increasingly turn to low-cost easily accessible AI chatbots for health advice, real-world evidence is needed on how patient-facing AI performs compared with PHC physicians. Objective To compare patient-facing AI chatbots with PHC physicians in diagnostic accuracy, clinical safety, and patient-centered communication using standardized patients in real-world PHC settings. Design, Setting, and Participants This standardized patient field experiment was conducted from February 6 to February 27, 2025, in 62 PHC institutions in China. Standardized patients (SPs) presented the same cases of unstable angina and asthma to 248 PHC physicians and to 2 widely used AI chatbots, ChatGPT-4o and DeepSeek-R1, on the same day. The analysis included 744 SP encounters. Exposures Consultation with a PHC physician, ChatGPT-4o, or DeepSeek-R1 for the same conditions. Main Outcomes and Measures Primary outcomes included diagnostic accuracy (correct diagnosis and correct medication), clinical safety (number and rate of unnecessary tests and medication prescribing), and patient-centeredness (range 1–4, with higher scores indicating more patient-centered communications). Results Among 248 PHC physicians, 66.9% were male and the mean (SD) age was 44.7 (9.4) years. Compared with PHC physicians, ChatGPT-4o and DeepSeek-R1 had higher rates of correct diagnosis (94.0% and 99.2% vs 30.6%; P<.001), correct medication recommendation (100.0% and 100.0% vs 35.4%; P<.001), and higher patient-centeredness scores (3.28 and 3.35 vs 2.55; P<.001), particularly in understanding the whole person and finding the common ground . However, both chatbots ordered more tests and recommended more medications than physicians, resulting in higher rates of unnecessary testing (90.3% and 96.4% vs 33.1%; P<.001) and inappropriate medication recommendations (73.4% and 62.1% vs 23.4%; P<.001). Findings were robust in multivariable regression models and when local physician performance was benchmarked against a nationally representative sample of PHC providers in China. Conclusions and Relevance In this field experiment, patient-facing AI chatbots had higher diagnostic accuracy and patient-centered communication scores than PHC physicians, but were substantially more likely to recommend unnecessary tests and inappropriate medications. These findings suggest that patient-facing AI may improve access to diagnostic and communicative support in under-resourced primary care, while also creating a risk of persuasive, AI-induced low-value care. Governance of patient-facing AI should therefore focus not only on diagnostic accuracy, but also on preventing excessive recommendations that may shape patient expectations before clinical encounters.

Source: Research Square

DOI: 10.21203/rs.3.rs-9988886/v1