Year: 2025
Type: conference-paper
Abstract: Scaling pre-training compute has proven effective for achieving multilinguality, but does the same hold for test-time scaling?In this work, we introduce MCLM, a multilingual math benchmark featuring competition-level problems in 55 languages.We test three test-time scaling methods-Outcome Reward Modeling (ORM), Process Reward Modeling (PRM), and Budget Forcing (BF)-on both Qwen2.5-1.5BMath and MR1-1.5B, a multilingual LLM we trained for extended reasoning.Our experiments show that using Qwen2.5-1.5BMath with ORM achieves a score of 35.8 on MCLM, while BF on MR1-1.5Battains 35.2.Although "thinking LLMs" have recently garnered significant attention, we find that their performance is comparable to traditional scaling methods like best-of-N once constrained to similar levels of inference FLOPs.Moreover, while BF yields a 20-point improvement on English AIME, it provides only a 1.94-point average gain across other languages-a pattern consistent across the other test-time scaling methods we studied-highlighting that test-time scaling may not generalize as effectively to multilingual tasks.To foster further research, we release MCLM, MR1-1.5B, and evaluation results.
Authors: Guijin Son, Jiwoo Hong, Hyunwoo Ko, James Thorne
Language: English
Field: Decision Sciences
Domain: Social Sciences
Open Access status: hybrid
OpenAlex ID: W4412945328