Work

Lightweight open-source large language models versus cTAKES for information extraction from discharge summaries: tobacco smoking status test case

Year: 2025

Type: article

Abstract: Objectives: To compare lightweight open-source large language models (LLMs) with cTAKES, a state-of-the-art natural language processing (NLP) system, in an information extraction task from hospitalization discharge summaries. Materials and Methods: Two readers annotated 250 randomly sampled adult discharge summaries (BJC HealthCare, 2018-2023) for tobacco smoking status as "Smoker," "Never smoker," "Unknown." Six LLMs (Llama-3 [1B-70B], gpt-oss-20B, MedGemma-27B) and cTAKES extracted smoking status from summaries. Performance was benchmarked against consensus annotations using weighted F1-score, macro F1-score, and per-class F1-scores and a noninferiority test. Results: < .021). Discussion: The high accuracy and efficiency of gpt-oss-20B support its potential as a practical, open-source alternative to traditional NLP for clinical information extraction. Conclusion: Lightweight LLMs can be applied for use across diverse clinical information extraction tasks without the need for task-specific fine-tuning.

Source: JAMIA Open

DOI: 10.1093/jamiaopen/ooaf182

Authors: David M Dávila-García, Matthew J Schuelke, Adam B Wilcox

Institution: Washington University in St. Louis

Language: English

Topic: Biomedical Text Mining and Ontologies

Subfield: Molecular Biology

Field: Biochemistry, Genetics and Molecular Biology

Domain: Life Sciences

SDG: Good health and well-being

Open Access status: gold

Estimated APC paid: $3,762