Work

Evaluating retrieval-augmented large language models for pediatric cardiology knowledge using standardized questions

Year: 2026

Type: article

Abstract: Large language models (LLMs) may support education for medical trainees, but their knowledge and safety in pediatric subspecialties remain underexplored. We developed NadasGPT, an interactive pediatric cardiology chatbot that integrates multiple frontier LLMs with an optional retrieval-augmented generation (RAG) pipeline grounded in a single pediatric cardiology textbook. We evaluated performance of eight LLMs on 304 board-style pediatric cardiology multiple-choice questions (MCQs) across medical student, resident, and fellow levels, comparing models with and without RAG. Baseline performance improved substantially across successive model generations, with gpt-5.2 and gemini-3.1-pro achieving 98.4% accuracy without RAG. Enabling a single-textbook RAG pipeline did not significantly change performance for any model except gpt-5.1 on fellow-level questions ( p = 0.016). Overall, as LLM performance approaches ceiling levels on standardized pediatric cardiology MCQs, a single-source RAG pipeline provides limited additional improvement in accuracy, underscoring the need to better characterize if and how RAG contributes to educational utility.

Source: npj Digital Medicine

DOI: 10.1038/s41746-026-03153-9

Authors: Jacob Weiser, John K. Triedman, Joshua Mayourian

Institutions: Boston Children's Hospital, Harvard University

Language: English

Topic: Artificial Intelligence in Healthcare and Education

Subfield: Health Informatics

Field: Medicine

Domain: Health Sciences

SDGs: Good health and well-being, Quality education

Open Access status: gold

Estimated APC paid: $4,090