Work

Benchmarking large language models for adverse drug reaction extraction in social media and clinical texts

Year: 2025

Type: article

Abstract: Adverse Drug Reactions represent a significant threat to patient safety and remain a critical focus of pharmacovigilance, yet traditional extraction methods struggle with ambiguity, under-reporting, and noisy biomedical or social media texts. Large Language Models (LLMs) have emerged as a promising alternative due to their contextual understanding and biomedical pre-training, but their reliability in distinguishing true ADRs from spurious associations is still uncertain. This study aims to systematically evaluate the effectiveness of state-of-the-art open- and closed-source LLMs, including both general-purpose and domain-specific models, for ADR extraction. We benchmark GPT-3.5-turbo, GPT-4o-mini, Microsoft Phi-3-mini, LLaMA-3.2-1B, DeepSeek, and BioMistral-7B using two complementary approaches: fine-tuning and in-context learning (zero-shot, one-shot, and few-shot). Experiments are conducted on two benchmark datasets, CADEC and SMM4H, with evaluation based on strict and relaxed precision, recall, and F1-score metrics. Results show that GPT-4o-mini achieves the highest fine-tuning performance, with relaxed F1-scores of 79.06% on CADEC and 66.80% on SMM4H. In the five-shot in-context learning setting, it achieves 65.57% on CADEC and 49.34% on SMM4H, highlighting both progress and ongoing limitations. These findings demonstrate that while LLMs hold significant promise for enhancing ADR extraction, they still face challenges in capturing the complexity and variability of real-world ADR expressions, indicating that further model adaptations and domain-specific refinements are required to improve their effectiveness in pharmacovigilance applications. • The study evaluates state-of-the-art open- and closed-source LLMs for ADR extraction. • Fine-tuning and in-context learning are compared on CADEC and SMM4H datasets. • GPT-4o-mini fine-tuning achieves the best performance with the highest relaxed F1-scores. • Results reveal progress but highlight challenges in handling noisy real-world texts. • Findings emphasize the need for domain-specific adaptations to enhance pharmacovigilance.

Source: Results in Engineering

DOI: 10.1016/j.rineng.2025.107362

Authors: Oumayma Elbiach, Hanane Grissette, El Habib Nfaoui

Institution: Sidi Mohamed Ben Abdellah University

Language: English

Topic: Pharmacovigilance and Adverse Drug Reactions

Subfield: Toxicology

Field: Pharmacology, Toxicology and Pharmaceutics

Domain: Life Sciences

SDG: Good health and well-being

Open Access status: gold

Estimated APC paid: $1,940