Work

(Meta-) evaluation of machine translation

Year: 2007

Type: conference-paper

Abstract: This paper evaluates the translation quality of machine translation systems for 8 language pairs: translating French, German, Spanish, and Czech to English and back. We carried out an extensive human evaluation which allowed us not only to rank the different MT systems, but also to perform higher-level analysis of the evaluation process. We measured timing and intra- and inter-annotator agreement for three types of subjective evaluation. We measured the correlation of automatic evaluation metrics with human judgments. This meta-evaluation reveals surprising facts about the most commonly used methodologies.

Source: Conference on Machine Translation (WMT)

DOI: 10.3115/1626355.1626373

Authors: Chris Callison-Burch, Cameron Fordyce, Philipp Koehn, Christof Monz, Josh Schroeder

Institutions: Johns Hopkins University, University of Edinburgh, University of London

Language: English

Topic: Natural Language Processing Techniques

Subfield: Artificial Intelligence

Field: Computer Science

Domain: Physical Sciences

Open Access status: bronze

Funders: European Commission, Defense Advanced Research Projects Agency, Advanced Research Projects Agency

Awards: G2515541506, G7463329482