Work

Large Language Model-Assisted Systematic Review: Validation Based on Cochrane Review Data

Year: 2025

Type: review

Abstract: Large Language Models (LLMs) offer potential for automating systematic reviews, a labor-intensive process in evidence-based medicine. We evaluated GPT-4o, GPT-4o-mini, and Llama 3.1:8B on abstract screening and risk of bias assessment using 12 Cochrane drug intervention reviews. GPT-4o achieved the best screening performance (recall 0.894, precision 0.492). We propose a one-shot inclusivity adjustment method enabling threshold modulation without repeated inferences. For risk of bias, accuracy varied by domain, highest in random sequence generation (0.873), and lowest in selective reporting (0.418). Our findings demonstrate LLMs' practical utility and current limitations in automating systematic reviews.

Source: Studies in health technology and informatics

DOI: 10.3233/shti250501

Authors: Siun Kim, Hyung-Jin Yoon

Institutions: Seoul National University Hospital, Seoul National University Hospital Biomedical Research Institute, Seoul National University

Language: English

Topic: Meta-analysis and systematic reviews

Subfield: Statistics, Probability and Uncertainty

Field: Decision Sciences

Domain: Social Sciences

SDG: Good health and well-being

Open Access status: hybrid

Funder: Korea Health Industry Development Institute