Agreeability testing of AMSTAR-PF, a tool for quality appraisal of systematic reviews of prognostic factor studies
Year: 2025
Type: preprint
Abstract: Background This paper details initial testing of the agreeability and usability of a novel quality appraisal tool for systematic reviews of prognostic factor studies: AMSTAR-PF. Methods Fourteen appraisers each assessed eight systematic reviews using AMSTAR-PF. Their ratings for each question and each article were compared, with interrater, inter-pair and intrapair agreeability calculated using Gwet’s agreement coefficient. Time of use and time to reach consensus were also recorded. Results Interrater agreement averaged 0.59 (range, 0.21-0.90), inter-pair 0.61 (range 0.24-0.91) and intrapair 0.75 (range 0.45-0.95) across the domains, with agreement for the overall rating 0.46 (95%CI 0.30-0.62) for interrater, 0.46 (95%CI 0.17-0.74) for inter-pair, and 0.68 (range of averages 0.22-1.00) for intrapair agreement. The majority (60.7%) of intrapair ratings were identical, with 94.6% of final ratings either identical or only one category different for the overall appraisal. The time taken to appraise a study with AMSTAR-PF improved with use and averaged around 34 minutes after the first two appraisals. Conclusions Despite some variance in agreeability for different domains and between different appraisers, the testing results suggest that AMSTAR-PF has clear utility for appraising the quality of systematic reviews of prognostic factor studies.
Source: medRxiv
Authors: ML Henry, NE O’Connell, RD Riley, KGM Moons, BJ Shea, L Hooft, SB Wallwork, JAA Damen, N Skoetz, RP Appiah, C Berryman, SM Crouch, GA Ferencz, AR Grant, KM Henry, AM Herman, EL Karran, I Koralegedera, HB Leake, E MacIntyre, B Mouatt, K Phuentsho, DA Van Der Laan, E Welsby, LK Wiles, EM Wilkinson, MK Wilson, MV Wilson, GL Moseley
Language: English
Subfield: Statistics, Probability and Uncertainty
Field: Decision Sciences
Domain: Social Sciences
Open Access status: green
OpenAlex ID: W4409425199