How to Estimate Intraclass Correlation Coefficients for Interrater Reliability from Planned Incomplete Data
Year: 2025
Type: article
Abstract: The interrater reliability (IRR) of observational data is often estimated by means of intraclass correlation coefficients (ICCs), which are flexible IRR estimators that are based on the variance decomposition of scores obtained by observations. ICCs are typically estimated using mean squares from an ANOVA model, the computation of which is not straightforward for incomplete data. However, many studies in behavioral research use planned missing observational designs, in which the raters partially vary across subjects. Planned missing designs result in incomplete data. Therefore, we simulated planned incomplete data and compared the computational accuracy (bias of point estimates, bias of variability estimates, root mean squared error, and coverage rates) and computational feasibility (convergence rates and estimation time) of three recently proposed estimation methods for ICCs: Markov chain Monte Carlo estimation of Bayesian hierarchical linear models, maximum likelihood estimation of random-effects models, and maximum likelihood estimation of common-factor models. Maximum likelihood estimation of random-effects models with Monte-Carlo confidence intervals is preferred based on all criteria. This article is accompanied by R code, which enables researchers to apply these estimation methods. A demonstration of the R code to a real-data set from an educational context is provided.
Source: Multivariate Behavioral Research
Institutions: Vrije Universiteit Amsterdam, University of Amsterdam
Language: English
Subfield: Statistics, Probability and Uncertainty
Field: Decision Sciences
Domain: Social Sciences
Open Access status: hybrid
OpenAlex ID: W4411390013