Coefficient of agreement between two raters corrected for category prevalence: Alternative to kappa.
Year: 2025
Type: article
Abstract: Cohen's kappa coefficient was introduced as a statistical measure to evaluate the degree of interrater agreement between two raters who classify each subject using categorical scales. Cohen posited that a certain level of agreement between raters is expected to occur by chance, and thus, kappa is designed to account for this expected chance agreement by adjusting the observed percent agreement. However, over time, several paradoxes and limitations have emerged in its interpretation, largely due to the underlying assumption of random chance agreement and its estimation. In this article, we propose that a portion of the observed percent agreement can be attributed to the interaction between category prevalence and the inherent characteristics of the categories themselves, such as their appeal, ambiguity, social desirability, or other factors related to the traits being measured. This prevalence-agreement effect can either positively or negatively influence the observed percent agreement. By moving away from the assumption of random assignment by raters, we derive a new coefficient of agreement that effectively removes the prevalence-agreement effect. We also discuss the significance of this new coefficient, its interpretation, and the stability of its estimation (standard error). (PsycInfo Database Record (c) 2025 APA, all rights reserved).
Source: Psychological Methods
DOI: 10.1037/met0000732
Author: Rashid Saif Almehrizi
Institution: Sultan Qaboos University
Language: English
Subfield: Statistics, Probability and Uncertainty
Field: Decision Sciences
Domain: Social Sciences
Open Access status: hybrid
OpenAlex ID: W4413375486