Noise or additional information? Leveraging crowdsource annotation item agreement for natural language tasks.
Year: 2015
Type: conference-paper
Abstract: In order to reduce noise in training data, most natural language crowdsourcing annotation tasks gather redundant labels and aggregate them into an integrated label, which is provided to the classifier.However, aggregation discards potentially useful information from linguistically ambiguous instances.For five natural language tasks, we pass item agreement on to the task classifier via soft labeling and low-agreement filtering of the training dataset.We find a statistically significant benefit from low item agreement training filtering in four of our five tasks, and no systematic benefit from soft labeling.
DOI: 10.18653/v1/d15-1035
Authors: Emily Jamison, Iryna Gurevych
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: hybrid
Funder: Volkswagen Foundation
Awards: G348198870, G6467963393
OpenAlex ID: W2250600206
