Metric Inversion in Generative AI Translation: Divergence Between Automated Evaluation Metrics and Human Expert Judgement in the LRM Era
Year: 2026
Type: conference-paper
Abstract: The paradigm of computer-assisted translation (CAT) is currently navigating a period of profound structural disruption fueled by Generative Artificial Intelligence (GAI). The transition from deterministic Neural Machine Translation (NMT) to reasoning-driven Large Reasoning Models (LRMs), such as DeepSeek R1, has introduced complex chain-of-thought (CoT) capabilities. However, this advancement has precipitated a “Metric Inversion” crisis. Traditional automated evaluation metrics, specifically BLEU and chrF, rely on surface-level n-gram overlap against a static human reference. As AI capability shifts from literal sequence mapping to creative semantic restructuring, these legacy benchmarks systemically penalize the exact linguistic and idiomatic qualities that human experts value most. Through a randomized double-blind study (N = 111), we contrast the performance of an LRM, a general LLM, and a standard NMT system. Results demonstrate that while human experts strongly favor LRM outputs for their superior fluency (Mean=8.81) and idiomaticity (Mean=8.61), automated metrics exhibit a significant and misleading bias toward traditional NMT (BLEU=0.258). Correlation analysis reveals a profound negative relationship (ρ = −0.401, p <.001) between BLEU scores and expert fluency ratings. This research highlights the urgent need for a paradigm shift from reference-dependent overlap matching to semantic-aware evaluation frameworks to preserve the structural integrity of AI-assisted education.
Authors: Yaobang Zhang, Yiming Zhang
Institution: Tianjin University of Technology
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
SDG: Quality education
Open Access status: hybrid
OpenAlex ID: W7213161033
