Work

A Measure of the System Dependence of Automated Metrics

Year: 2025

Type: conference-paper

Abstract: Automated metrics for Machine Translation have made significant progress, with the goal of replacing expensive and time-consuming human evaluations.These metrics are typically assessed by their correlation with human judgments, which captures the monotonic relationship between human and metric scores.However, we argue that it is equally important to ensure that metrics treat all systems fairly and consistently.In this paper, we introduce a method to evaluate this aspect.

Source: Annual Meeting of the Association for Computational Linguistics (ACL)

DOI: 10.18653/v1/2025.acl-short.8

Authors: Pius Von Däniken, Jan Milan Deriu, Mark Cieliebak

Institution: ZHAW Zurich University of Applied Sciences

Language: English

Topic: Natural Language Processing Techniques

Subfield: Artificial Intelligence

Field: Computer Science

Domain: Physical Sciences

Open Access status: hybrid

Funders: National Science Foundation, Schweizerischer Nationalfonds zur Förderung der Wissenschaftlichen Forschung

OpenAlex ID: W4412889505