Year: 2026
Type: conference-paper
Abstract: Machine translation (MT) evaluation is central in guiding researchers on how to improve a model's performance.Current automatic evaluation practices fail to provide reliable insights into the specific translation errors that occur, especially for low-resource languages.This paper introduces the Lux-MT-Test-Suite, enabling a linguistically motivated and fine-grained analysis of Luxembourgish-English (LB-EN) MT based on 896 test items covering 12 linguistic categories and 36 linguistic phenomena.We compare a baseline local LLM (GEMMA 3), its fine-tuned counterpart (LUXMT), and a proprietary state-of-the-art LLM (GPT-5) to analyse what local LLMs learn through fine-tuning in a low-resource setting and to assess performance differences between local and proprietary systems.The findings identify specific performance gains through fine-tuning, minor degradations, a difference in translation strategies, performance gaps between local and proprietary models, and remaining challenges.
Author: Nils Rehlinger
Institution: University of Luxembourg
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: gold
OpenAlex ID: W7140131734
FWCI:
Cites:
Cited by:
Related to: