Year: 2011
Type: article
Abstract: Recently heard at a tutorial in our field: "It cost me less than one hundred bucks to annotate this using Amazon Mechanical Turk!" Assertions like this are increasingly common, but we believe they should not be stated so proudly; they ignore the ethical consequences of using MTurk (Amazon Mechanical Turk) as a source of labour.Manually annotating corpora or manually developing any other linguistic resource, such as a set of judgments about system outputs, represents such a high cost that many researchers are looking for alternative solutions to the standard approach.MTurk is becoming a popular one.However, as in any scientific endeavor involving humans, there is an unspoken ethical dimension involved in resource construction and system evaluation, and this is especially true of MTurk.We would like here to raise some questions about the use of MTurk.To do so, we will define precisely what MTurk is and what it is not, highlighting the issues raised by the system.We hope that this will point out opportunities for our community to deliberately value ethics above cost savings.
Source: Computational Linguistics
DOI: 10.1162/coli_a_00057
Authors: Karën Fort, Gilles Adda, K. Bretonnel Cohen
Institutions: Centre National de la Recherche Scientifique, Institut de l'Information Scientifique et Technique, Université Sorbonne Paris Nord, Laboratoire d'Informatique de Paris-Nord, Université Paris-Sud, Laboratoire d'Informatique pour la Mécanique et les Sciences de l'Ingénieur, University of Colorado Boulder, University of Colorado Anschutz
Language: English
Subfield: Computer Science Applications
Field: Computer Science
Domain: Physical Sciences
Open Access status: bronze
OpenAlex ID: W2116705992
