Evaluation of large language models for clinical decision support in prostate cancer: a multidisciplinary Delphi study
Year: 2026
Type: article
Abstract: Aim: Large language models (LLMs) may help interpret clinical guideline content. However, their performance in guideline-based prostate cancer (PC) scenarios remains insufficiently validated. This study evaluated the agreement between a multidisciplinary expert panel and responses generated by three leading LLMs (Claude, Gemini and ChatGPT) when addressing structured, guideline-derived clinical questions on PC. Materials & methods: A multidisciplinary tumor board created 25 structured questions based on the 2025 European Association of Urology guidelines. The guideline figures and tables were transformed into plain-language summaries for use by the LLMs. Each model was prompted with a simple, unmodified query and a panel of experts in PC management independently rated their level of agreement with the responses. The evaluation used a two-round modified Delphi method with predefined consensus criteria. Results: Eleven experts evaluated 25 guideline-based questions. After two Delphi rounds, consensus in agreement was reached for 23/25 responses (92%) with Gemini, 20/25 (80%) with ChatGPT and 14/25 (56%) with Claude (p = 0.0033). Explicit disagreement was noted in three questions (12%) with Claude after the second round. Median scores of at least 7 out of 9, indicating strong agreement, were achieved in 24/25 responses (96%) for Gemini, 22/25 (88%) for ChatGPT, and 14/25 (56%) for Claude (p = 0.0002). Most consensus difficulties involved complex cases, advanced imaging, genetic testing or questions with limited data or controversial topics. Conclusion: Advanced LLMs, when provided with PC guidelines with plain-language adaptations of figures and tables, can achieve strong alignment with expert consensus in routine situations. However, performance varies among models, with Claude showing lower agreement rates, especially in complex cases. These results emphasize the importance of expert oversight, continuous validation and standardized guideline formats interpretable by both clinicians and AI systems.
Institutions: Universidad de Zaragoza, Hospital Universitario Miguel Servet, Instituto de Investigación Sanitaria Aragón, Hospital Universitario Puerta del Mar, Instituto de Investigación Biosanitaria de Granada, Hospital Universitario Virgen de las Nieves, Hospital Universitario La Paz, Instituto de Investigación Biomédica de A Coruña, Instituto Ramón y Cajal de Investigación Sanitaria, Hospital Universitario Ramón y Cajal, Hospital Regional Universitario de Málaga, Hospital Clínico San Carlos, Clínica Girona, Complejo Hospitalario Universitario de Santiago
Language: English
Subfield: Health Informatics
Field: Medicine
Domain: Health Sciences
Open Access status: diamond
OpenAlex ID: W7225253151