Work

Higher inter-rater agreement but more low-risk judgments: the influence of rater experience on RoB 2 assessments

Year: 2026

Type: article

Abstract: It remains unclear whether risk-of-bias judgments made using the revised Cochrane risk-of-bias tool for randomized trials (RoB 2) by less experienced raters, even after standard training, systematically differ from those made by experienced raters. Our study recruited 12 inexperienced and 6 experienced raters to assess the risk of bias in 94 results from 37 randomized controlled trials using RoB 2. After the independent assessments were completed, raters were randomly paired within groups to reach consensus judgments. We calculated Gwet’s agreement coefficient 2 to evaluate inter-rater reliability in both phases. After exclusions, 10 inexperienced raters (four pairs) and 6 experienced raters (three pairs) were included. Inexperienced raters demonstrated higher inter-rater agreement across multiple domains in both independent and consensus assessments, whereas experienced raters showed higher agreement only in the “randomization process” domain. Despite this higher agreement, inexperienced raters were more likely to assign “low risk” ratings than experienced raters across most domains. After consensus, agreement in the “randomization process” domain improved in the experienced group, whereas it increased in two domains but decreased in two in the inexperienced group. Overall, rating distributions remained stable among experienced raters but varied across several domains among inexperienced raters. This may suggest that increased reliability among inexperienced raters may reflect systematic differences in interpretation rather than more accurate assessments. These findings highlight potential challenges in applying RoB 2 among less experienced users and underscore the importance of adequate training and calibration.

Source: BMC Medical Research Methodology

DOI: 10.1186/s12874-026-03007-8

Authors: Linghui Xiang, Yinyan Gao, Xuemei Sun, Betty Huan Wang, Zhichao Gong, Mengshi Chen, Irene Xinyin Wu

Institutions: Central South University, Chinese University of Hong Kong, Hunan University of Traditional Chinese Medicine, Second Affiliated Hospital of Hunan University of Traditional Chinese Medicine

Language: English

Topic: Meta-analysis and systematic reviews

Subfield: Statistics, Probability and Uncertainty

Field: Decision Sciences

Domain: Social Sciences

Open Access status: gold

Estimated APC paid: $3,090

OpenAlex ID: W7214066101

FWCI:

Cites:

Cited by:

Related to: