Clinician expertise and prompt engineering enhance cancer information extraction in electronic health records by small language models
Year: 2026
Type: article
Abstract: Real-world data (RWD) in unstructured electronic health records (EHRs) is crucial for understanding complex diseases like cancer, but extracting structured information is challenging due to linguistic variability, semantic complexity, and privacy concerns. This study evaluates the performance of four small, locally deployable language models for information extraction from Italian EHRs. We examine three prompting strategies (zero-shot, few-shot, and annotated few-shot) across English and Italian, involving clinicians with varying expertise to assess the impact of prompt design on accuracy. We evaluate the performance of four open-source small language models (SLMs) for clinical information extraction from Italian electronic health records (EHRs) in the APOLLO 11 trial on non-small cell lung cancer (NSCLC). The extraction protocol involves four steps: problem definition, data preprocessing, Large Language Model (LLM)-based information extraction, and output evaluation. We show that general-purpose models (e.g., LLaMA 3.1 8B) outperform biomedical models in most tasks, particularly in extracting binary features. Multiclass variables such as TNM (Tumor, Node, Metastasis) staging, PD-L1 (Programmed death-ligand 1), and ECOG-PS (Eastern Cooperative Oncology Group-Performance Status) are more difficult due to implicit language and lack of standardization. Few-shot prompting and native-language inputs significantly improve performance and reduced hallucinations. Clinical expertise enhances consistency in the extraction, particularly among students using annotated examples. The study confirms that privacy-preserving SLMs can be deployed locally for efficient and secure cancer data extraction. Findings highlight the need for hybrid systems combining SLMs with expert input and underline the importance of aligning clinical documentation practices with SLM capabilities. This is the first study to benchmark SLMs on Italian EHRs and investigate the role of clinical expertise in prompt engineering, offering valuable insights for the future integration of SLMs into real-world clinical workflows. This study aimed to explore how a type of computational models called small language models (SLMs) can help extract important information from patients’ medical records. We tested four different models, three different ways to prompt the models to analyse the medical records and records in English and Italian. We found some models were better at extracting information than others. As far as we are aware, this is the first study to benchmark SLMs on Italian EHRs and investigate the role of clinical expertise in prompt engineering, offering valuable insights for the future integration of SLMs into real-world clinical workflows. Corso et al. evaluate small, locally deployable language models for clinical information extraction from Italian Electronic Health Records. General purpose models outperform biomedical models and combining few-shot prompting with clinical expertise enhances accuracy and reduces hallucinations.
Source: Communications Medicine
Authors: Federica Corso, Vittoria Peppoloni, Laura Mazzeo, Giuseppe Leone, Luana Passos, Vanja Mišković, Justin Armanini, Alberto Ferrarin, Isabella Catharina Wiest, Fabian Wolf, Giulia Montelatici, Rebecca Romanò, Paolo Ambrosini, Tommaso Capoccia, Stefano Natangelo, Simone Rota, Paola Andena, Marta De Ponti, Alessandra Russo, Giulia Stasi, Leonardo Provenzano, Andrea Spagnoletti, Marco Meazza Prina, Chiara Cavalli, Claudia Giani, Roberta Serino, Michele Borracino, Chiara Bonalume, Rosa Maria Di Mauro, Claudia Agosta, Andra Diana Dumitrascu, Giorgia Di Liberti, Giulia Corrao, Teresa Beninato, Monica Ganzinelli, Mario Occhipinti, Marta Brambilla, Claudia Proto, Jakob Nikolas Kather, Alessandra Laura Giulia Pedrocchi, Filippo De Braud, Giuseppe Lo Russo, Paolo Baili, Arsela Prelaj
Institutions: Fondazione IRCCS Istituto Nazionale dei Tumori, Politecnico di Milano, University of Minho, University of Milano-Bicocca, University Hospital Carl Gustav Carus, Else Kröner Fresenius Center for Digital Health, Technische Universität Dresden, University of Milan, University of Leeds, University Hospital Heidelberg, National Center for Tumor Diseases
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: gold
Estimated APC paid: $3,690
Funders: National Institute for Health and Care Research, Department of Health and Social Care, European Commission, Deutscher Akademischer Austauschdienst, Deutsche Forschungsgemeinschaft, Ministero dell’Istruzione, dell’Università e della Ricerca, Deutsche Krebshilfe, Gemeinsame Bundesausschuss, National Institutes of Health, HORIZON EUROPE Framework Programme, Leeds Biomedical Research Centre
Awards: G1387481293, G162135912, G1949475607, G2654538971, G2820917408, Understanding Gene ENvironment Interaction in ALcohol-related hepatocellular carcinoma, G3848825097, Open Consortium for Decentralized Medical Artificial Intelligence, G463921964, G4712284355, G4785148476, G4876319204, G4881028819, G4996692071, G6266899304, G6697820746, G9000156230, G9000314692
OpenAlex ID: W7168482985