Work

Comparative Evaluation of Large Language Models and Human Specialists in Acid-Base Disorder Interpretation and Sepsis Management: A Prospective Simulation Study (Preprint)

Year: 2026
Type: preprint
Abstract: BACKGROUND Large language models (LLMs) such as ChatGPT and Google Gemini have demonstrated promising capabilities in medical reasoning and clinical decision support. However, their comparative performance against human specialists in critical care scenarios, particularly acid-base disorder interpretation and sepsis management, remains inadequately characterized. OBJECTIVE This study aimed to compare the diagnostic and therapeutic decision-making performance of advanced AI models (ChatGPT-4 and Google Gemini), a consensus-based ensemble AI approach, and human medical specialists in acid-base disorder interpretation and sepsis management scenarios using validated clinical vignettes. METHODS A total of 45 clinical case vignettes (20 acid-base disorder cases and 25 sepsis management cases) were developed by an expert panel. Cases were independently evaluated by 20 human specialists (10 emergency medicine physicians and 10 anesthesiologists), ChatGPT-4, Google Gemini, and a simple majority-voting ensemble model. Blinded evaluation was ensured throughout. Performance metrics included diagnostic accuracy, treatment recommendation appropriateness, and Surviving Sepsis Campaign (SSC) bundle compliance rates. RESULTS For acid-base disorder interpretation, the ensemble AI model achieved the highest overall accuracy (86.0%), followed by anesthesiologists (84.5%), ChatGPT-4 (83.7%), emergency physicians (83.2%), and Google Gemini (79.5%). In simple metabolic and respiratory disorders, AI models demonstrated comparable or superior performance to human experts (>90% accuracy). However, human specialists outperformed individual AI models in mixed acid-base disorders (humans: 75.5% vs ChatGPT: 68.5%, Gemini: 65.3%, P<.05). For sepsis management, SSC hour-1 bundle compliance was highest in the ensemble model (95.8%), followed by ChatGPT-4 (94.2%), human experts (91.5%), and Gemini (89.7%). CONCLUSIONS Advanced LLMs demonstrate comparable performance to human specialists in straightforward acid-base and sepsis scenarios, with ensemble approaches showing potential for improved accuracy. However, human expertise remains superior in complex, atypical presentations requiring nuanced clinical judgment. These findings are limited to text-based simulations and require validation in real-world clinical environments.
Language: English
Field: Medicine
Open Access status: green
OpenAlex ID: W7117972613
Cites:
Cited by:
Related to: 10