Evaluating Large Language Model Diagnostic Classification in Neuropsychology
Year: 2026
Type: preprint
Abstract: Objective: Large language models (LLMs) have demonstrated promise in healthcare for clinical documentation and decision support. Much of the information in electronic medical records, however, is structured or clinician-synthesized documentation. There is less certainty about the capability of LLMs in devising clinically appropriate diagnostic hypotheses from unstructured data such as a patient interview.Method: Deidentified clinical interview transcripts from 32 older adults referred for neuropsychological evaluation of memory complaints were submitted to 10 commercially available LLMs using an institutionally approved platform with a standardized prompt. Models classified each case using four categories: normal, mild cognitive impairment (MCI), dementia, or uncertain. Model classifications were compared with clinician reference diagnoses based on neuropsychological evaluation. Results: Model-inferred and reference diagnoses showed exact agreement rates from 18.8% to 56.2% with poor to slight chance-corrected agreement (Cohen’s κ = −.178 to .196). Classification errors across models were more often overclassification rather than underclassification. Using dichotomous cognitive status (i.e., normal versus impaired), seven of ten models showed sensitivity over 90% for identifying impairment, but specificity ranged from 0 to 60%. Workflow response time varied by model and was not associated with accuracy. Conclusions: Using unstructured and unannotated clinical interview data, general-purpose LLMs showed limited, model-dependent agreement with clinician diagnoses and a tendency to overclassify impairment. LLMs warrant further evaluation as tools that support clinical workflows, with revised prompting strategies and systematic efforts to improve classification.
Source: PsyArXiv (OSF Preprints)
Language: English
Subfield: Health Informatics
Field: Medicine
Domain: Health Sciences
Open Access status: green
OpenAlex ID: W7220659096