Evaluator-Masked Assessment of 4 Consumer-Facing Large Language Models on Expert Anterior Segment Ophthalmology Questions
Year: 2026
Type: article
Abstract: Objective: To compare expert-rated quality and safety of responses from four contemporary large language models (LLMs)-ChatGPT, Gemini, Claude, and Grok-on expert-level anterior segment ophthalmology questions. Design: Cross-sectional, masked, randomized-display-order, multievaluator comparative study. Participants: Eleven fellowship-trained cornea and anterior segment subspecialists. Methods: Forty-five expert-level questions spanning six anterior segment domains were administered to each LLM under zero-shot conditions in a single prespecified session (January 19, 2026), with memory, browsing, and retrieval disabled. Responses were masked to model identity and independently rated on five 5-point Likert metrics (expert-rated accuracy, completeness/clinical utility, clarity/readability, overall safety, and potential for harm). No specialist-authored reference answer key was used; all metrics index expert-rated perception rather than verified clinical correctness. Main Outcome Measures: Median [interquartile range], mean ± standard deviation, favorable-rating proportion (≤2) with 95% bootstrap confidence interval, Friedman omnibus with Kendall's W, and Bonferroni-adjusted pairwise Wilcoxon tests with rank-biserial correlation. Prespecified sensitivity analyses included length-adjusted mixed models and a masking-integrity diagnostic; post hoc analyses added question-level severe-safety and style-restricted comparisons. Results: < 0.001), persisting when two-evaluator concurrence was required. Ordinal Krippendorff's α was 0.021-0.168; the 11-evaluator mean used as the primary analytic unit showed moderate reliability (intraclass correlation coefficient [ICC(2,k)] = 0.58-0.69 for quality metrics). Conclusions: Claude and Grok received more favorable expert-rated quality than ChatGPT and Gemini. Post hoc, a larger severe-safety-rating tail was identified for these two models despite no significant difference in mean safety; this signal is hypothesis-generating. The findings reflect expert preference, not validated clinical correctness, at one time point. Financial Disclosures: The authors have no proprietary or commercial interest in any materials discussed in this article.
Source: Ophthalmology Science
Authors: Soonwon Yang, Hyoji Han, Hyun Soo Lee
Institution: Catholic University of Korea
Language: English
Subfield: Health Informatics
Field: Medicine
Domain: Health Sciences
Open Access status: gold
Estimated APC paid: $2,500
OpenAlex ID: W7208757176