Benchmarking large language models for biomedical natural language processing applications and recommendations
Year: 2025
Type: article
Abstract: The rapid growth of biomedical literature poses challenges for manual knowledge curation and synthesis. Biomedical Natural Language Processing (BioNLP) automates the process. While Large Language Models (LLMs) have shown promise in general domains, their effectiveness in BioNLP tasks remains unclear due to limited benchmarks and practical guidelines. We perform a systematic evaluation of four LLMs-GPT and LLaMA representatives-on 12 BioNLP benchmarks across six applications. We compare their zero-shot, few-shot, and fine-tuning performance with traditional fine-tuning of BERT or BART models. We examine inconsistencies, missing information, hallucinations, and perform cost analysis. Here we show that traditional fine-tuning outperforms zero- or few-shot LLMs in most tasks. However, closed-source LLMs like GPT-4 excel in reasoning-related tasks such as medical question answering. Open-source LLMs still require fine-tuning to close performance gaps. We find issues like missing information and hallucinations in LLM outputs. These results offer practical insights for applying LLMs in BioNLP.
Source: Nature Communications
Language: English
Subfield: Molecular Biology
Domain: Life Sciences
Open Access status: gold
Estimated APC paid: $6,990
Awards: Finding combinatorial drug repositioning therapy for Alzheimer's disease and related dementias, G5850682332, G6148477407, G7767218177, A Translational Informatics Framework to Mine Efficacy and Safety of Dietary Supplements, Addressing Factual Inaccuracy and Unfaithful Reasoning of Large Language Models in Biomedicine and Healthcare, Natural language processing and medical imaging analysis for multi-modality computer assisted diagnosis of ophthalmic diseases, Detecting synergistic effects of pharmacological and non-pharmacological interventions for AD/ADRD
OpenAlex ID: W4409190866