Work

Artificial intelligence versus physician decision-making in acute heart failure: a comparative evaluation of four contemporary AI systems

Year: 2026

Type: article

Abstract: Aims Large language models (LLMs) are increasingly used by the public and healthcare professionals for clinical reasoning. However, they were developed as general-purpose language models rather than dedicated clinical decision-support tools, and their performance in real-world acute heart failure (AHF) management remains insufficiently evaluated. We compared four contemporary AI models with physician decision-making. Methods and results This single-centre retrospective observational study incorporated a cross-sectional AI evaluation of consecutive patients treated at Sotiria General Hospital between July and November 2025, excluding acute coronary syndrome. ChatGPT, ChatGPT Plus, DeepSeek, and an ESC guideline-based retrieval-augmented generation (RAG) model independently assessed identical standardized cases. Recommendations were compared with physician decisions on diagnosis, treatment, guideline-directed medical therapy, timing, and prognosis using Cohen’s kappa, McNemar’s test, Cochran’s Q, Wilcoxon signed-rank testing, and multivariable logistic regression. Diagnostic performance differed significantly across systems (Cochran’s Q = 8.477, P = 0.037). In the balanced diagnostic cohort, ChatGPT Plus achieved the highest heart failure (HF) accuracy (accuracy 86.5%, sensitivity 87.0%, and specificity 86.0%), followed by RAG (accuracy 77.5%), ChatGPT (accuracy 77.0%), and DeepSeek (accuracy 69.0%). For HF within the top three differential diagnoses, sensitivity was 90.0–97.0%; specificity was 70.0% for ChatGPT Plus, 68.0% for RAG, 19.0% for DeepSeek, and 15.0% for ChatGPT. Physician agreement was highest for guideline-directed therapies, with greater variability in prognosis and escalation of care. RAG more closely matched physician therapeutic timing but did not consistently outperform general-purpose models. Conclusion Contemporary AI models show promise in AHF decision support but substantial task-specific variability, supporting use as complements rather than replacements for physician judgement.

Source: European Heart Journal - Digital Health

DOI: 10.1093/ehjdh/ztag169