Year: 2025
Type: preprint
Abstract: Because unreliable measurement can lead to lower statistical power, attenuated effect sizes and residual confounding, reliability should be routinely reported. However, estimating reliability is challenging for complex cognitive and behavioural assessments, because test-retest data are often unavailable and most single-administration methods are designed for fixed-item questionnaires. We introduce relative measurement uncertainty (RMU), a Bayesian procedure for estimating reliability that applies to any measurement model with subject-level parameters, including computational models. RMU draws pairs of samples from each subject's posterior and calculates the correlation between draws. We show analytically that the RMU point estimate converges to classical test theory reliability for a simple multilevel model. We provide an R function for computing RMU and illustrate how to apply it to mean score and reinforcement learning models fitted using the R packages brms and hBayesDM, respectively. In simulations, RMU provided accurate reliability estimates and well-calibrated uncertainty intervals across linear factor, signal detection and reinforcement learning models. RMU was also more accurate (lower root mean square error) than existing estimators (coefficient alpha, coefficient H, split-half reliability). However, RMU's primary advantage is that it offers a general method for estimating reliability and its uncertainty across the broad range of models that Bayesian methods can fit.
Language: English
Field: Decision Sciences
Domain: Social Sciences
Open Access status: gold
Awards: G1762083400, G5275711260, The Intergenerational Transmission of Mental Health Problems: Identifying Causal Pathways and How to Interrupt Them, G581313191, Cambridge Service for Data Driven Discovery (CSD3) - A National Data Intensive Science Cloud for Converged Simulation, AI & Analytics
OpenAlex ID: W4414629582