Work

Findings of the WMT26 Multilingual Instruction Shared Task: Small Models Are Not Yet Polyglots

Year: 2026

Type: conference-paper

Abstract: We present the findings of the WMT26 Multilingual Instruction Shared Task (MIST), evaluating small (≤10B parameter) open-weight language models across 24 languages on three subtasks: context-based QA, cross-lingual summarization, and open-ended generation.Submissions from eight teams explore fine-tuning, knowledge distillation, and prompt engineering.Our findings demonstrate that small models are not yet polyglots: while leading systems perform well on high-resource languages, performance drops sharply on low-resource varieties, and comprehension degrades significantly when transferring across non-English language pairs.In open-ended generation, human evaluation and automatic rule-based verification exhibit strong overall rank correlation (ρ=0.79)but diverge locally at the top: human evaluators favor distilled pipelines with superior fluency and cross-lingual equity, whereas automatic verifiers reward concise promptengineered systems.Furthermore, multi-stage pipelines suffer from prompt-language inertia on cross-lingual instructions, and summarization exhibits severe English leakage unless penalized by language gating.Finally, while baseline fluency is largely achieved by modern small models, multi-constraint instruction following and unanswerable question refusal remain the primary performance bottlenecks.We release all test sets, system outputs, human judgments, and evaluation code under a permissive license.

Source: Conference on Machine Translation (WMT)

DOI: 10.18653/2026.wmt-main.50