Work

Automated Interpretation of EEG Reports Using a Large Language Model with Structured Confidence Outputs

Year: 2026

Type: preprint

Abstract: Objective To evaluate a large language model (LLM) pipeline for extracting structured diagnostic labels and confidence levels from unstructured free-text EEG reports, addressing the barrier of narrative data analysis. Methods We developed a hierarchical schema classifying reports for four abnormality types using a four-point confidence scale. Two certified EEG technologists annotated the reports independently; one was designated the Reference Annotator (RA) and the other the Second Annotator (SA). We implemented a grammar-constrained Mistral-7B pipeline and evaluated its agreement with the RA alongside the SA and classical NLP baselines using Core Agreement and Certainty-Adjusted Agreement. A portability analysis held the task, labels, output grammar, report identifiers, reference annotations, and endpoints fixed while a MedGemma configuration used its model-declared instruction-turn format. Results For overall abnormality, Mistral-7B achieved 96% Core Agreement with the RA and the SA achieved 98%. Mistral-7B exceeded the tested low-resource base-lines in most categories and identified rare epileptiform abnormalities for which the baselines produced no or few true positives. Among reports with matching overall-abnormality decisions, Conditional Certainty Agreement was about 80% for Mistral–RA on each evaluated report set and 90% and 94% for SA–RA. The MedGemma configuration showed higher agreement than Mistral-7B in several non-epileptiform categories, but focal epileptiform performance involved a trade-off between missed and additional positive calls. Conclusions Under the studied report-level conditions, the grammar-constrained pipeline produced structured annotations with high binary agreement to the RA. After adapting the prompt presentation to its model-declared instruction-turn format, the same structured task and evaluation framework transferred to a newer local model. The results support its use as a research annotation method; they do not establish diagnostic ground truth, external validation, or readiness for clinical deployment. Significance This study demonstrates a locally executable method for converting de-identified EEG narratives into structured research annotations while retaining an ordinal representation of annotator certainty.

Source: medRxiv

DOI: 10.64898/2026.07.07.26357190