4CPS-159 Comparative evaluation of ChatGPT-4o and ChatGPT-3.5 in clinical rule-guided dose interventions in hospitalised patients with renal dysfunction
Year: 2025
Type: conference-paper
Abstract: Background and Importance Clinical decision support systems (CDSS) play a crucial role in identifying medications that require dose modifications for patients with renal impairment. ChatGPT has the potential to be integrated into electronic health record (EHR) systems for providing such dosing recommendations. Two recent studies, however showed that the performance of ChatGPT-3.5 in this domain was suboptimal.1 2 Recently, ChatGPT-4o was released, offering more contextually relevant responses compared to previous versions. In this study, we want to compare the performance of ChatGPT-4o with version 3.5 in clinical rule-guided dose interventions for hospitalised patients with renal impairment. Aim and Objectives To evaluate the effectiveness of ChatGPT-4o compared to ChatGPT-3.5 in providing clinical rule-guided dose interventions for patients with renal impairment. Material and Methods This cross-sectional study was conducted at a teaching hospital in Europe. CDSS alerts related to renal dysfunction were collected from the EHR over a two-week period and presented to both ChatGPT versions and an expert panel. Alerts were presented with and without patient-specific variables, and suggested medication interventions were compared to assess performance. Results A total of 172 CDSS alerts were generated for 80 patients. For alerts without patient variables, ChatGPT-4o provided ‘correct and identical’ responses to 41.3% of the alerts, ‘correct but different’ responses to 18.6%, and ‘incorrect’ responses to 40.1%. In contrast, ChatGPT-3.5 provided ‘correct and identical’ responses for only 19.9% of the alerts, ‘correct but different’ responses for 26.7%, and ‘incorrect’ responses for 53.4%. When alerts included patient-specific variables, ChatGPT-4o offered ‘correct and identical’ responses in 39.0% of cases, ‘correct but different’ responses in 12.2%, and ‘incorrect’ responses in 48.8%. In comparison, ChatGPT-3.5 provided ‘correct and identical’ responses for only 16.7%, ‘correct but different’ for 16.0%, and ‘incorrect’ responses for 67.3% of the alerts. Notably, both versions of ChatGPT showed better accuracy for newer drugs, such as direct oral anticoagulants. Conclusion and Relevance ChatGPT-4o demonstrated significantly better performance compared to ChatGPT-3.5 in clinical rule-guided dose interventions for patients with renal dysfunction, showing an improvement in accuracy and a reduction in incorrect recommendations. However, despite these improvements, ChatGPT-4o is insufficient for automatic integration into EHR systems to manage CDSS alerts for renal dysfunction. References and/or Acknowledgements https://ejhp.bmj.com/content/early/2023/05/31/ejhpharm-2023-003750.long https://link.springer.com/article/10.1007/s00228-024-03687-5 Conflict of Interest No conflict of interest
Authors: P Van Der Linden, M Van Nuland
Institution: Tergooi
Language: English
Subfield: Health Informatics
Field: Medicine
Domain: Health Sciences
Open Access status: gold
OpenAlex ID: W4408376807