Year: 2025
Type: article
Abstract: Disclosure: S. Sidhu: None. G. Wu: None. A. Sidhu: None. E. Pan: None. A. Ashok: None. D. Kim: None. M. Del Buono: None. Background: Ozempic use in the United States saw an 83.9% per-month increase rate between 2019 and 2022. GLP-1 analogs are currently being researched for their involvement in increasing the risk of developing cancer, such as thyroid cancer. One study found that while the consistent use of GLP-1 receptor agonists lowered the incidence of prostate and lung cancer development, it significantly increased the risk of thyroid cancer. Purpose: Evaluate whether AI models can correctly identify possible cancer risks associated with GLP-1 agonist drugs. Methods: Five questions were asked to Claude, Cohere, Gemini, ChatGPT 4o Mini, and ChatGPT 4o. Textual responses from these large language models (LLMS) were recorded and scored on a scale of 1 to 5, both manually and by the chatbots themselves. NMDS clustering analysis was performed on the data with the following measurements: Word Count, Average Sentence Length, Number of Paragraphs, Manual Score, and AI Score. The parameters were scaled and mapped onto plots using the Bray distance formula. Results: Gemini exhibited a linear distribution near the top of the plot, and its distribution contained several outliers. Claude and Cohere also showed several outliers. ChatGPT 4o and 4o mini showed no clusters, and ChatGPT 4o Mini’s even spray pattern suggests that while it is consistent, it also generates more varied and flexible outputs. Claude’s main distribution showed a very tight cluster, which could indicate high consistency and similarity in output format. Conclusion: Overall, the data shows that some chatbots, such as Gemini and ChatGPT 4o mini, are consistent in their responses, while others are more varied, although consistency did not necessarily correspond to accuracy. The presence of outliers in LLMs such as Cohere and Claude suggests differences in the quality of datasets they are being trained on and highlights the need for more robust datasets so patients can receive high-quality responses that do not differ between chatbots. Presentation: Sunday, July 13, 2025
Source: Journal of the Endocrine Society
Authors: Sahej Sidhu, Gloria Wu, Aadjot Sidhu, Ethan Pan, Aarabhi Ashok, Doyoung Kim, Milan Del Buono
Language: English
Subfield: Health Informatics
Field: Medicine
Domain: Health Sciences
Open Access status: gold
Estimated APC paid: $3,255
OpenAlex ID: W7093301145