Year: 2026
Type: article
Abstract: Dear Editors Sciurti and colleagues recently reported in this journal a careful evaluation of three compact large language models, GPT-4o mini, Llama 3.1 8B, and Gemma 2 9B, applied to title and abstract screening across three published systematic reviews on vaccine literacy, antibiotic-resistant Acinetobacter baumannii, and vitamin supplementation in COVID-19. 1 The three corpora differ in size, topic, and prevalence of eligible records, and the authors evaluated each model individually as well as a majorityvoting ensemble at predefined rating thresholds.The headline finding, and the one most likely to shape practice, is the estimated workload reduction of 79% to 91% achieved by the ensemble at a 50-rating threshold across the three reviews, while sensitivity remained high.Compact models that run on local hardware at low cost, with savings of this magnitude, would be a meaningful addition to the systematic reviewer's toolkit.This figure deserves close reading, however.It is a retrospective summary computed against groundtruth labels that already exist in the published reviews, and it describes the model's classification behaviour rather than the time a review team will save when deploying the workflow prospectively.The two are related, but they are not the same quantity.Whether the retrospective figure transfers to a real workflow depends on how the team chooses to handle the records the model excluded, and that choice is not visible in the metric itself.The metric used by Sciurti et al., defined as the proportion of records correctly excluded by the model out of the total, follows the convention proposed by Syriani et al. and is now widely adopted in this literature.2 As a property of the classifier, it is well defined and easy to compute.The risk is that, when reported as "workload saving", it invites a reading it cannot quite support.A practitioner encountering "79% to 91% workload saving" may reasonably infer that, by adopting the workflow, the team will only need to screen the remaining 9% to 21% of citations, on the grounds that the model has correctly removed the rest.The arithmetic of the metric encourages this reading, since saved and remaining work appear to add up to one.Yet the inference rests on a step the metric does not justify, namely the step of treating the model's excluded set as work that no longer needs to be done.Systematic reviews are designed to identify, as completely as possible, all studies that meet the eligibility criteria, and methodological guidance treats missing an eligible study as a more serious error than retaining an irrelevant one.Sciurti et al. acknowledge this asymmetry explicitly when they argue that sensitivity should be prioritised and imprecision tolerated.The implication, however, runs further
Source: Research Synthesis Methods
Authors: Honghao Lai, Jiayi Liu, Long Ge, Janne Estill
Institutions: Lanzhou University, University of Geneva
Language: English
Subfield: Statistics, Probability and Uncertainty
Field: Decision Sciences
Domain: Social Sciences
Open Access status: hybrid
Estimated APC paid: $3,020
OpenAlex ID: W7204467859