Comparative Performance of Artificial Intelligence Models and Human Experts in Acid-Base Disorder Interpretation and Sepsis Management (Preprint)
Year: 2025
Type: preprint
Abstract: BACKGROUND Large language models (LLMs) such as ChatGPT and Google Gemini have demonstrated promising capabilities in medical reasoning and clinical decision support. However, their comparative performance against human specialists in critical care scenarios, particularly acid-base disorder interpretation and sepsis management, remains inadequately characterized. OBJECTIVE This study aimed to compare the diagnostic and therapeutic decision-making performance of advanced AI models (ChatGPT-4 and Google Gemini), an ensemble AI approach, and human medical specialists in acid-base disorder interpretation and sepsis management scenarios. METHODS A total of 45 clinical cases (20 acid-base disorder cases and 25 sepsis management cases) were developed by an expert panel consisting of one anesthesiologist, one emergency medicine specialist, and one intensivist. Cases were independently evaluated by 20 human specialists (10 emergency medicine physicians and 10 anesthesiologists), ChatGPT-4, Google Gemini, and an ensemble model combining both AI outputs. Sepsis cases were designed according to Surviving Sepsis Campaign (SSC) guidelines. Performance metrics included diagnostic accuracy, treatment recommendation appropriateness, and SSC bundle compliance rates. RESULTS For acid-base disorder interpretation, the ensemble AI model achieved the highest overall accuracy (86.0%), followed by anesthesiologists (84.5%), ChatGPT-4 (83.7%), emergency physicians (83.2%), and Google Gemini (79.5%). In simple metabolic and respiratory disorders, AI models demonstrated comparable or superior performance to human experts (>90% accuracy). However, human specialists outperformed individual AI models in mixed acid-base disorders (humans: 75.5% vs ChatGPT: 68.5%, Gemini: 65.3%, P<.05). For sepsis management, SSC hour-1 bundle compliance was highest in the ensemble model (95.8%), followed by ChatGPT-4 (94.2%), human experts (91.5%), and Gemini (89.7%). The ensemble model demonstrated superior performance in complex septic shock scenarios (AUC=0.891, 95% CI 0.862-0.920) compared to human experts (AUC=0.847, 95% CI 0.812-0.882, P=.023). CONCLUSIONS Advanced LLMs demonstrate comparable performance to human specialists in straightforward acid-base and sepsis scenarios, with ensemble approaches showing potential for improved accuracy. However, human expertise remains superior in complex, atypical presentations requiring nuanced clinical judgment. A hybrid human-AI approach may optimize clinical decision-making in critical care settings.
Authors: Evren Ekingen, Mete Ucdal
Language: English
Subfield: Health Informatics
Field: Medicine
Domain: Health Sciences
Open Access status: green
OpenAlex ID: W4417340134
