Work

Form and Meaning in Intrinsic Multilingual Evaluations

Year: 2026

Type: conference-paper

Abstract: Intrinsic evaluation metrics for conditional language models, such as perplexity or bits-percharacter, are widely used in both mono-and multilingual settings.These metrics are rather straightforward to use and compare in monolingual setups, but rest on a number of assumptions in multilingual setups.One such assumption is that comparing the perplexity of CLMs on parallel sentences is indicative of their quality since the information content (here understood as the semantic meaning) is the same.However, the metrics are inherently measuring information content in the informationtheoretic sense.Consistency in meaning does not neutralize different forms (paraphrases).We make such assumptions explicit and discuss their implications.We perform experiments with six metrics on two multi-parallel corpora both with mono-and multilingual models.We find that current metrics are not universally comparable and look at the form-meaning debate to provide some explanation for this.

Source: Conference of the European Chapter of the Association for Computational Linguistics

DOI: 10.18653/v1/2026.eacl-long.113

Authors: Wessel Poelman, Miryam de Lhoneux

Institution: KU Leuven

Language: English

Topic: Natural Language Processing Techniques

Subfield: Artificial Intelligence

Field: Computer Science

Domain: Physical Sciences

Open Access status: hybrid

Funders: Vlaams Supercomputer Centrum, Fonds Wetenschappelijk Onderzoek, KU Leuven, Vlaamse regering

Award: Fairness in Multilingual NLP