Work

Combining Crowd and Expert Labels Using Decision Theoretic Active Learning

Year: 2015
Type: conference-paper
Abstract: We consider a finite-pool data categorization scenario which requires exhaustively classifying a given set of examples with a limited budget. We adopt a hybrid human-machine approach which blends automatic machine learning with human labeling across a tiered workforce composed of domain experts and crowd workers. To effectively achieve high-accuracy labels over the instances in the pool at minimal cost, we develop a novel approach based on decision-theoretic active learning. On the important task of biomedical citation screening for systematic reviews, results on real data show that our method achieves consistent improvements over baseline strategies. To foster further research by others, we have made our data available online.
Authors: An Nguyen, Byron Wallace, Matthew Lease
Language: English
Open Access status: diamond
OpenAlex ID: W2174985112
FWCI: 5.057
Cites: 34
Cited by: 82
Related to: 10