Year: 2025
Type: article
Abstract: Reviews are considered a cornerstone of evidence-based research, supporting innovative approaches across various disciplines. However, the title and abstract screening process is resource-intensive. The increasing volume of published studies exacerbates this challenge, often leading to delayed publication of results. Machine learning tools offer potential solutions and reducing screening time. However, reliable stop criteria for terminating the screening process without missing relevant studies remain unclear. This study aims to evaluate different stop criteria within ASReview to optimize the efficiency of systematic reviews. A total of 35,000 simulations of title and abstract screenings were conducted using ASReview across five different datasets from the SYNERGY database. Three stopping criteria found in literature were tested: Stopping (1) when a pre-calculated number of relevant studies is reached, (2) after a consecutive series of irrelevant studies, and (3) after screening a predefined percentage of the dataset. The accuracy and efficiency of these stop criteria were analyzed by comparing the retrieved relevant studies with the total number of relevant studies in the dataset. All tested stop criteria reduced screening time. However, none of the criteria proved universally reliable. The first criterion failed in 98% of simulations, either overestimating or underestimating the necessary screening effort. The second criterion exhibited significant variability across datasets, with no consistent threshold applicable to all cases. The third criterion showed potential but was dataset-dependent, making generalizability challenging. While ASReview significantly reduces screening time, no stop criterion was consistently reliable across datasets. The study highlights the need for adaptive, data-driven stopping rules tailored to specific review contexts. Future research should explore hybrid approaches combining multiple stopping rules and validation strategies. Key messages • No universal stop criterion reliably optimizes systematic review screening in ASReview. • Adaptive, data-driven stop criteria are necessary to balance efficiency and accuracy in systematic reviews.
Authors: C Kempny, K Annac, D Wahidie, Y Yilmaz-Aslan, P Brzoska
Institution: Witten/Herdecke University
Language: English
Subfield: Statistics, Probability and Uncertainty
Field: Decision Sciences
Domain: Social Sciences
Open Access status: gold
Estimated APC paid: $3,181
OpenAlex ID: W7101444053