Diagnostic Agreement Between a General-Purpose AI Model and Retinal Specialists in Color Fundus Photography—A Pilot Study
Year: 2026
Type: article
Study design: Observational Study
Abstract: Background: Artificial intelligence (AI) has shown strong performance in disease-specific retinal screening tasks; however, its reliability in heterogeneous clinical diagnostic settings remains unclear. This study compared a general-purpose multimodal AI model with experienced retinal specialists in the interpretation of color fundus photographs (CFPs). Methods: In this pilot retrospective cross-sectional study, 66 CFPs were independently evaluated by a masked retinal specialist and an AI model (Google Gemini 2.5 Flash). Diagnoses were compared with those of the unblinded treating specialist. The comparison was inherently asymmetric, as the reference specialist had access to full clinical information, whereas the masked evaluators performed image-only assessment. Agreement was assessed using weighted percent agreement and Gwet’s AC2 with quadratic weights. Results: Substantial agreement was observed between the two human specialists (AC2 = 0.67). In contrast, agreement between the AI model and the reference specialist was low (AC2 = −0.58). Direct comparison between the masked specialist and the AI also showed limited reliability (AC2 = −0.38). Conclusions: In this pilot study, the evaluated AI model demonstrated limited agreement relative to a context-informed specialist reference. These findings support cautious interpretation of consumer-facing multimodal AI in open-ended retinal image assessment and warrant validation in larger, multicenter studies.
Source: Journal of Clinical Medicine
DOI: 10.3390/jcm15093430
Language: English
Topic: Retinal Imaging and Analysis
Subfield: Radiology, Nuclear Medicine and Imaging
Field: Medicine
Domain: Health Sciences
Open Access status: gold
Estimated APC paid: $3,137
OpenAlex ID: W7160136286