Year: 2009
Type: conference-paper
Abstract: Manual evaluation of translation quality is generally thought to be excessively time consuming and expensive.We explore a fast and inexpensive way of doing it using Amazon's Mechanical Turk to pay small sums to a large number of non-expert annotators.For $10 we redundantly recreate judgments from a WMT08 translation task.We find that when combined non-expert judgments have a high-level of agreement with the existing gold-standard judgments of machine translation quality, and correlate more strongly with expert judgments than Bleu does.We go on to show that Mechanical Turk can be used to calculate human-mediated translation edit rate (HTER), to conduct reading comprehension experiments with machine translation, and to create high quality reference translations.
Author: Chris Callison-Burch
Institution: Johns Hopkins University
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: bronze
OpenAlex ID: W2143539737