Expert appraisal of AI-generated recommendations in cardiovascular surgical decision-making: a multisurgeon study
Year: 2026
Type: article
Abstract: Background Large Language Models (LLMs) have demonstrated strong performance in standardized medical examinations; however, their reliability in real-world surgical decision-making, particularly in complex and guideline-ambiguous clinical scenarios, remains uncertain. Cardiovascular surgery, characterized by high risk, complex comorbidities, and frequent reliance on individualized clinical judgment, represents a critical domain for examining how artificial intelligence (AI)–generated recommendations are appraised by clinicians operating in high-risk, judgment-intensive settings. Accordingly, this study aimed to present an expert appraisal of the perceived clinical appropriateness of GPT-5.2–generated recommendations as evaluated by cardiovascular surgeons, and to examine whether appraisal varies according to surgeon experience and scenario type, including guideline-ambiguous (“grey-zone”) decision-making contexts. Methods Fifty original cardiovascular surgical scenarios—covering coronary, valvular, aortic, and peripheral vascular diseases—were developed, including approximately 20% predefined guideline-ambiguous (“grey-zone”) cases. All scenarios were processed using GPT-5.2 with a standardized system instruction and deterministic settings (temperature 0.0; top_p 1.0), with web browsing enabled; each scenario was run in a separate session to minimize contextual carryover. Fourteen actively practicing cardiovascular surgeons independently evaluated the AI-generated recommendations using an anchored 5-point Likert scale capturing perceived clinical appropriateness rather than objective correctness or patient-level outcomes, with scores of 4–5 prespecified as indicating clinically appropriate management. Surgeons were stratified into Early-Career (1–9 years), Mid-Career (10–19 years), and Senior (≥ 20 years) groups. Differences in ratings were analyzed using linear mixed-effects models accounting for repeated measures and cross-classified clustering at both surgeon and scenario levels. Results Across 700 individual evaluations, AI-generated recommendations received a high overall mean rating of 4.31 ± 0.8, with 86% of responses classified as clinically appropriate (Likert score 4–5). Ratings did not differ significantly between Early- and Mid-Career surgeons (p = 0.750), whereas Senior surgeons assigned lower scores than the Mid-Career group (mean difference − 0.27, p = 0.001). Grey-zone scenarios received lower ratings overall (mean difference − 0.19, p = 0.023). Sensitivity analyses using ordinal proportional odds regression yielded directionally consistent results. Conclusions In this surgeon-based appraisal study, AI-generated recommendations demonstrated high perceived clinical appropriateness in cardiovascular surgical decision-making. Differences in ratings reflected an experience-dependent acceptability threshold, alongside an additive scenario-type effect, without evidence of Experience × ScenarioType interaction. These findings suggest that large language models may function as complementary decision-support resources in cardiovascular surgery, while underscoring the continued importance of expert judgment in complex and individualized clinical scenarios.
Source: Scientific Reports
Language: English
Subfield: Health Informatics
Field: Medicine
Domain: Health Sciences
Open Access status: gold
Estimated APC paid: $2,690
OpenAlex ID: W7212050062