Work

Embarrassingly Easy Document-Level MT Metrics: How to Convert Any Pretrained Metric into a Document-Level Metric

Year: 2022

Type: conference-paper

Abstract: We present a very simple method for extending pretrained machine translation metrics to incorporate document-level context.We apply our method to four popular metrics: BERTScore, Prism, COMET, and the reference-free metric COMET-QE.We evaluate our documentlevel metrics on the MQM annotations from the WMT 2021 metrics shared task and find that the document-level metrics outperform their sentence-level counterparts in about 85% of the tested conditions, when excluding results on low-quality human references.Additionally, we show that our document-level extension of COMET-QE dramatically improves accuracy on discourse phenomena tasks, supporting our hypothesis that our document-level metrics are resolving ambiguities in the reference sentence by using additional context.

Source: Conference on Machine Translation (WMT)

DOI: 10.18653/v1/2022.wmt-1.6

Authors: Giorgos Vernikos, Brian Thompson, Prashant Mathur, Marcello Federico

Institution: Amazon (United States)

Language: English

Topic: Natural Language Processing Techniques

Subfield: Artificial Intelligence

Field: Computer Science

Domain: Physical Sciences

Open Access status: hybrid

OpenAlex ID: W7126413112

FWCI: 1.52

Cites:

Cited by: 13

Related to: