Large language models for systematic review tasks compared with Cochrane reviews: a protocol for evaluating accuracy, agreement, and time efficiency
Year: 2026
Type: article
Study design: Study Protocol
Abstract: Large language models (LLMs) show promise for accelerating systematic review (SR) stages, yet comprehensive comparative evaluations against validated standards remain lacking. To compare the accuracy, agreement, and time efficiency of four LLMs (GPT, Claude, Gemini, and DeepSeek) across five SR stages—search strategy formulation, deduplication, article selection, risk of bias assessment, and evidence synthesis—against published Cochrane Reviews as the reference standard. Comparative study with randomized assignment using a Williams Latin square design. Twenty five Cochrane SRs of interventions (2020–2025) will be evaluated by 15 researchers stratified by experience. Prompts will be developed using the 4D framework; the underlying process model will be validated via Delphi consensus, and resulting prompt wording via expert content review. Outcomes include recall, sensitivity, Cohen’s kappa, and execution time. Ethical exemption will be requested. Results will be disseminated through five peer-reviewed publications.
Source: Systematic Reviews
Authors: Josue Rivadeneira, Luis Fuenmayor-González, Daniel Peñaherrera-Vásquez, Carlos Eduardo Duque Martínez, María Fernanda García, Juan Pablo Holguín, Tamara Otzen, Graziela De Luca Canto, Virginia Fernandes Moça Trevisani, Aline Rocha, Franz Eduardo Del Pozo Sanchez, Fabio Mendes Torres, Carlos Manterola
Language: English
Subfield: Statistics, Probability and Uncertainty
Field: Decision Sciences
Domain: Social Sciences
Open Access status: gold
Estimated APC paid: $3,090
OpenAlex ID: W7204886924
FWCI:
Cites:
Cited by:
Related to: