Work

Accuracy and confidence calibration of large language models on neurology board-style questions

Year: 2026

Type: preprint

Abstract: ABSTRACT Large language models (LLMs) are increasingly evaluated for clinical tasks, but safe use requires reliable uncertainty estimates as well as high accuracy. We tested eight general-purpose LLMs on 100 validated neurology board-style multiple-choice questions and compared three confidence signals: self-reported confidence, critique-implied confidence from an independent model instance, and consistency-based confidence from repeated sampling. Accuracy varied from 45% (Llama 3.1 8B) to 92% (Gemini 3 Flash/Pro), but confidence was generally overestimated, particularly in lower-accuracy models. Consistency-based confidence provided the best discrimination between correct and incorrect answers (pooled AUROC 0.74), outperforming self-report, although calibration differences between methods were not significant. Calibration should therefore be reported alongside accuracy when clinical LLMs are evaluated. Short Description We evaluated the accuracy and confidence of eight large language models on 100 neurology board-style questions. We found substantial overconfidence, and differences between confidence methods in their ability to distinguish correct from incorrect answers.

Source: medRxiv

DOI: 10.64898/2026.10.17.26365807

Authors: Neil Graham, Stephen Auger, Bahaa Attaallah, Amit Batla, Gregory Scott

Institutions: UK Dementia Research Institute, University College London, Imperial College London, UCL Queen Square Institute of Neurology

Language: English

Topic: Artificial Intelligence in Healthcare and Education

Subfield: Health Informatics

Field: Medicine

Domain: Health Sciences

SDG: Good health and well-being

Open Access status: green

Funder: Multiple System Atrophy Trust