Work

To Re(label), or Not To Re(label)

Year: 2014
Type: conference-paper
Abstract: One of the most popular uses of crowdsourcing is to provide training data for supervised machine learning algorithms. Since human annotators often make errors, requesters commonly ask multiple workers to label each example. But is this strategy always the most cost effective use of crowdsourced workers? We argue "No" --- often classifiers can achieve higher accuracies when trained with noisy "unilabeled" data. However, in some cases relabeling is extremely important. We discuss three factors that may make relabeling an effective strategy: classifier expressiveness, worker accuracy, and budget.
Authors: Christopher Lin, . Mausam, Daniel Weld
Language: English
Open Access status: diamond
OpenAlex ID: W1617082848
FWCI: 3.259
Cites: 39
Cited by: 62
Related to: 10