Work

LLM-Assisted Annotation for Spanish Medical Corpus Construction: An Empirical Evaluation

Year: 2026
Type: article
Abstract: Named entity annotation in clinical text is a costly and time-intensive task, particularly for low-resource languages such as Spanish. This article introduces CLINFEM, an XML-annotated corpus comprising 1,200 Spanish medical reports from two regions of Spain. CLINFEM was created using a thorough human-in-the-loop annotation process combining large language model pre-annotation and systematic expert review. It is imperative to acknowledge that each of the 1,200 medical reports underwent meticulous review and correction by a trained human expert. Large language model pre-annotation was employed solely for the purpose of saving time and was never intended to serve as a substitute for manual annotation. Consequently, the entire CLINFEM corpus is a human-validated, gold-standard resource. GPT-4o and Gemini 1.5 Pro were evaluated as pre-annotators using a custom scheme comprising eight clinical entity types and one relational category. GPT-4o achieved a Cohen's κ of 0.73 against human annotation, while Gemini 1.5 Pro obtained κ = 0.41. Human expert review yielded an overall inter-annotator agreement of κ = 0.869 (F1 = 0.793), with a gold-standard subset agreement of κ = 0.9894. This approach reduced the overall annotation effort by around 74% compared to a fully manual approach (81% for the bulk phase alone), saving over 1,000 hours. Error rates varied by medical speciality, with oncology requiring the highest correction rate (94.3% of reports). These results demonstrate that using large language models to assist annotation is a viable and efficient strategy for constructing clinical corpora in Spanish, while also highlighting the continued necessity of human oversight.
Language: English
Field: Medicine
Open Access status: diamond
OpenAlex ID: W7236208308
FWCI:
Cites: 25
Cited by:
Related to: 10