Year: 2025
Type: preprint
Abstract: Expert judgement is inherently noisy, but the consequences of this variability are often underestimated. Here I examine the UK Research Excellence Framework (REF) as a case study illustrating how judgment noise can interact with structural features of a multi-stage evaluation system. Simulations show that noise in both internal departmental reviews and REF panel reviews, in conjunction with the bounded REF rating scale, produces systematic underestimation of the highest-performing departments. Panel noise, rather than internal noise, had a disproportionate impact on department rankings, with even moderate noise producing severe distortions. At a policy level, these results underscore the need for a noise audit of REF panel ratings. More broadly, the case of REF demonstrates that noise embedded in sequential evaluation, for example, in hiring and grant review decisions, does not “cancel out”, but instead can propagate in potentially surprising ways.
Author: Robert Ward
Institution: Bangor University
Language: English
Subfield: Statistics, Probability and Uncertainty
Field: Decision Sciences
Domain: Social Sciences
Open Access status: gold
OpenAlex ID: W4414496048
Cites:
Cited by:
Related to: