A comparative expert evaluation of ChatGPT and Gemini on urolithiasis: clinical accuracy versus citation reliability
Year: 2026
Type: article
Abstract: The rapid evolution of large language models (LLMs) presents opportunities and risks, including ‘hallucination’, for urological practice. This study compares the performance of ChatGPT 5.2 Pro and Gemini 3 Pro in answering urolithiasis queries, and evaluates the discrepancy between clinical accuracy and citation reliability. In this cross-sectional study conducted between January and February 2026, 65 clinical questions based on the 2025 European Association of Urology (EAU) guidelines were administered to both models. Two blinded, board-certified urologists independently evaluated the 130 generated responses. The outputs were scored on three separate dimensions. These were medical accuracy, guideline adherence and citation quality. Each dimension was scored on a 1–5 scale. Gemini 3 Pro demonstrated statistically higher scores compared to ChatGPT 5.2 Pro in terms of medical accuracy and guideline adherence (both p < 0.001). Gemini 3 Pro provided responses that were rated as clinically appropriate by experts at a rate of 98.5%, compared to 81.5% for GPT-5.2 Pro ( p = 0.001). However, both models had critically flawed citation reliability. At the overall response level, ChatGPT generated high-risk responses in 95.4% of cases, compared to 69.2% for Gemini ( p < 0.001). At the individual citation level, 45.8% of GPT-5.2 Pro’s citations and 24.8% of Gemini 3 Pro’s citations were entirely fabricated (score 1a). Meanwhile, 30.3% and 31.4% of citations, respectively, were real publications with incorrect or mismatched DOIs (score 1b). Notably, there was no statistical correlation between medical accuracy and citation quality (ChatGPT 5.2 Pro: rho = 0.033, p = 0.793; Gemini 3 Pro: rho=-0.195, p = 0.120). While Gemini 3 Pro provides highly accurate, guideline-adherent clinical information for urolithiasis, neither model currently serves as a reliable primary tool for academic literature retrieval. There is a stark discrepancy between the models’ high clinical accuracy and their critically poor citation reliability. Therefore, it is unsafe to rely on AI-generated medical content without strictly and independently verifying the underlying references for clinical practice.
Source: Scientific Reports
Authors: Yasin Aktaş, Adem Tunçekin
Institution: Usak University
Language: English
Subfield: Health Informatics
Field: Medicine
Domain: Health Sciences
Keywords: urolithiasis, large language models, ChatGPT, Gemini, hallucinations, citation reliability, clinical accuracy, guideline adherence, EAU guidelines, AI-generated medical content, reference verification, expert evaluation of AI responses, artificial intelligence in medical education, AI in clinical decision support
Open Access status: gold
Estimated APC paid: $2,690
OpenAlex ID: W7236238538
