Year: 2026
Type: conference-paper
Abstract: We introduce mmPISA-bench, a compact highquality multilingual reasoning benchmark derived from the OECD Programme for International Student Assessment (PISA).The benchmark consists of 25 multiple-choice questions that require reasoning in order to be answered correctly.Each question is provided in official human translations to 43 languages and complemented with machine-translated counterparts (i.e., 2,150 data points in total).We evaluate two mainstream proprietary LLMs across languages, reasoning effort levels, and translation types in terms of their ability to answer the questions correctly.Our results show that modern LLMs can reason effectively across all evaluated languages, achieve accuracy comparable to human test-takers, with some performance variations across covered languages.We further find that machinetranslated questions do not degrade accuracy relative to official human translations which suggests that high-quality machine translation (synthetic data) might often be adequate for large-scale multilingual reasoning evaluations where official translations are not available.Finally, we analyze token usage and related inference cost and find that LLMs usage in some languages is simultaneously more expensive and less accurate.
Authors: Yerzhan Sapenov, Jaromir Savelka
Institution: Carnegie Mellon University
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: hybrid
Funder: Accenture
OpenAlex ID: W7166840367