Year: 2025
Type: article
Abstract: Abs tractAim: Male circumcision remains routine in some countries for neonatal or religious reasons; however, it continues to be the subject of ongoing debate concerning its health benefits, potential risks, and implications for bodily autonomy.This study aims to evaluate the reliability of patient-facing content generated by four widely used large language models (LLMs) on various aspects of male circumcision.Methods: A search regarding LLMs was conducted using 20 standardized questions on 10 May 2025.Responses from ChatGPT, Copilot, Gemini, and Perplexity were evaluated by three independent experts.Inter-rater reliability was assessed with the intraclass correlation coefficient, and model performance differences were analyzed using Kruskal-Wallis tests with Bonferroni correction.Results: Inter-rater reliability was strong, with an intraclass correlation coefficient of 0.79 (p<0.001).Perplexity demonstrated statistically significant lower performance compared to ChatGPT, Copilot, and Gemini when evaluated across the thematic domains (p<0.001).Similarly, Perplexity performed statistically significantly worse than the other models across the criteria of clarity, structure, utility, and factual accuracy (p<0.001).Conclusion: Gemini and Copilot were the top performers across both thematic domains and evaluation criteria, highlighting substantial differences among LLMs in their ability to provide accurate and well-structured medical information regarding male circumcision.While ChatGPT shows promise for patient guidance, the inconsistent performance of models such as Perplexity highlights the need for cautious implementation and continuous oversight in healthcare communication.
Source: Medical Bulletin of Haseki
Authors: Ismail Ulus, Gokhan Ceker, Ibrahim Hacibey
Language: English
Subfield: Health Informatics
Field: Medicine
Domain: Health Sciences
Open Access status: diamond
OpenAlex ID: W4413812624