Year: 2025
Type: conference-paper
Abstract: Referred to as LLM-as-judge, a generative large language model (LLM) has demonstrated considerable efficacy as an evaluator in various tasks, including Machine Translation (LAJ-MT) by predicting scores or identifying error types for individual sentences.However, its dependability in practical application has yet to be demonstrated, as there is only an approximated match due to the task's open-ended nature.To address this problem, we introduce a straightforward and novel meta-evaluation strategy PROMPTCUE and evaluate cutting-edge LAJ-MT models such as GEMBA-MQM.We identify their fundamental deficits, including certain label biases and the inability to assess near-perfect translations.To improve reliability, we investigate more trustworthy and less biased models using multidimensional prompt engineering.Our findings indicate that the combination of span-level error quantification and a rubric-style prompt tailored to the characteristics of LLMs has efficiently addressed the majority of the challenges current LAJ-MT models face.Furthermore, it demonstrates a considerably enhanced alignment with human values.Accordingly, we present RUBRIC-MQM, the LAJ-MT for highend models and an updated version of GEMBA-MQM.
Author: Ahrii Kim
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: hybrid
OpenAlex ID: W4412886973