Work

Interrater disagreement resolution: A systematic procedure to reach consensus in annotation tasks

Year: 2021

Type: conference-paper

Abstract: We present a systematic procedure for interrater disagreement resolution. The procedure is general, but of particular use in multiple-annotator tasks geared towards ground truth construction. We motivate our proposal by arguing that, barring cases in which the researchers’ goal is to elicit different viewpoints, interrater disagreement is a sign of poor quality in the design or the description of a task. Consensus among annotators, we maintain, should be striven for, through a systematic procedure for disagreement resolution such as the one we describe.

Source: UvA-DARE (University of Amsterdam)

Authors: Yvette Oortwijn, Thijs Ossenkoppele, Arianna Betti

Language: English

Topic: Reliability and Agreement in Measurement

Subfield: Statistics, Probability and Uncertainty

Field: Decision Sciences

Domain: Social Sciences

Open Access status: green

Funders: Universiteit van Amsterdam, Nederlandse Organisatie voor Wetenschappelijk Onderzoek

Awards: Ideas at Scale - Towards a Computational History of Ideas (e-Ideas), Visual Analytics for the World's Library Data

OpenAlex ID: W3153733894

FWCI: 7.543

Cites: 33

Cited by: 10

Related to: 10