Work

EUROPA Eval: A Parallel, Inspectable Capability Benchmark for Post-Training in 36 European Languages

Year: 2026
Type: article
Abstract: Technical report, code, benchmark items and run artifacts for EUROPA Eval, an open multilingual capability benchmark for post-trained language models. It applies the same 12 task templates (reading, arithmetic, grounded answering and calibrated abstention, structured JSON, exact instruction following, tool-error recovery, a defined text calculation, summarization, translation, harmful-request refusal and revision tracking) to each of the 36 languages prioritized by OpenEuroLLM (432 items), with strict deterministic scoring and a pinned LLM judge. The report evaluates nine models, including five post-training variants of one 9B OpenEuroLLM base. The PDF is in paper/europa-eval.pdf.
Institution: AI Sweden
Language: English
Open Access status: green
OpenAlex ID: W7236231593
FWCI:
Cites: 1
Cited by:
Related to: 10