Systematic benchmarking demonstrates large language models have not reached the diagnostic accuracy of traditional rare-disease decision support tools
Year: 2026
Type: article
Abstract: Large language models (LLMs) show promise in supporting differential diagnosis, but their performance is challenging to evaluate due to the unstructured nature of their responses, and their accuracy compared to existing diagnostic tools is not well characterized. To assess the current capabilities of LLMs to diagnose genetic diseases, we benchmarked these models on 5213 previously published case reports using the Phenopacket Schema, the Human Phenotype Ontology and Mondo disease ontology. Prompts generated from each phenopacket were sent to seven LLMs, including four generalist models and three LLMs specialized for medical applications. The same phenopackets were used as input to a widely used diagnostic tool, Exomiser, in phenotype-only mode. The best LLM ranked the correct diagnosis first in 23.6% of cases, whereas Exomiser did so in 35.5% of cases. While the performance of LLMs for supporting differential diagnosis has been improving, it has not reached the level of commonly used traditional bioinformatics tools. Future research is needed to determine the best approach to incorporate LLMs into diagnostic pipelines.
Authors: Justin T. Reese, Leonardo Chimirri, Yasemin Bridges, Daniel Danis, J. Harry Caufield, Michael A. Gargano, Carlo Kroll, Andrew Schmeder, Fengchen Liu, Kyran Wissink, Julie A. McMurry, Adam S. L. Graefe, Enock Niyonkuru, Daniel R. Korn, Elena Casiraghi, Giorgio Valentini, Julius O. B. Jacobsen, Melissa Haendel, Damian Smedley, Christopher J. Mungall, Peter N. Robinson
Language: English
Subfield: Health Informatics
Field: Medicine
Domain: Health Sciences
Open Access status: hybrid
Estimated APC paid: $2,790
Funders: U.S. Department of Energy, Alexander von Humboldt-Stiftung, National Institutes of Health, Office of Science, National Human Genome Research Institute, National Institute of Child Health and Human Development, Basic Energy Sciences, Eunice Kennedy Shriver National Institute of Child Health and Human Development
Awards: G1286236842, G1286251239, G1586932880, G2618837358, G30714459, G3083819904, G3689434148, G382641700, G419881185, G42916606, G4501827968, G498139845, G650090426, G6652958823, A phenomics-first resource for interpretation of variants, G9000032241, G9000354844, Increasing the Yield and Utility of Pediatric Genomic Medicine with Exomiser, G969889393
OpenAlex ID: W7131249977