Popular large language model chatbots’ accuracy, comprehensiveness, and self-awareness in answering ocular symptom queries
Year: 2023
Type: article
Abstract: In light of growing interest in using emerging large language models (LLMs) for self-diagnosis, we systematically assessed the performance of ChatGPT-3.5, ChatGPT-4.0, and Google Bard in delivering proficient responses to 37 common inquiries regarding ocular symptoms. Responses were masked, randomly shuffled, and then graded by three consultant-level ophthalmologists for accuracy (poor, borderline, good) and comprehensiveness. Additionally, we evaluated the self-awareness capabilities (ability to self-check and self-correct) of the LLM-Chatbots. 89.2% of ChatGPT-4.0 responses were 'good'-rated, outperforming ChatGPT-3.5 (59.5%) and Google Bard (40.5%) significantly (all p < 0.001). All three LLM-Chatbots showed optimal mean comprehensiveness scores as well (ranging from 4.6 to 4.7 out of 5). However, they exhibited subpar to moderate self-awareness capabilities. Our study underscores the potential of ChatGPT-4.0 in delivering accurate and comprehensive responses to ocular symptom inquiries. Future rigorous validation of their performance is crucial to ensure their reliability and appropriateness for actual clinical use.
Source: iScience
Authors: Krithi Pushpanathan(Conceptualization, Data curation, Formal analysis, Investigation, Methodology, Validation, Visualization, Writing – original draft, Writing – review & editing), Zhi Wei Lim(Conceptualization, Data curation, Investigation, Methodology, Writing – original draft, Writing – review & editing), Samantha Min Er Yew(Conceptualization, Data curation, Investigation, Methodology, Validation, Writing – original draft, Writing – review & editing), David Ziyou Chen(Writing – review & editing), Hazel Anne Hui'En Lin(Writing – review & editing), Jocelyn Hui Lin Goh(Writing – review & editing), Wendy Meihua Wong(Writing – review & editing), Xiaofei Wang(Methodology, Writing – review & editing), Marcus Chun Jin Tan(Writing – review & editing), Victor Teck Chang Koh(Writing – review & editing), Yih-Chung Tham(Conceptualization, Data curation, Funding acquisition, Investigation, Methodology, Supervision, Validation, Writing – original draft, Writing – review & editing)
Language: English
Subfield: Health Informatics
Field: Medicine
Domain: Health Sciences
Open Access status: gold
Estimated APC paid: $3,150
Award: G30712838
OpenAlex ID: W4387496544
