Year: 2026
Type: conference-paper
Abstract: When asked a question in a language less seen in its training data, current reasoning large language models (RLMs) often exhibit dramatically lower performance than when asked the same question in English.In response, we introduce SP3F (Self-Play with Privileged Pairwise Feedback), a two-stage framework for enhancing multilingual reasoning without any data in the target language(s).First, we supervise fine-tune (SFT) on translated versions of English question-answer pairs to raise base model correctness.Second, we perform RL with feedback from a pairwise judge in a self-play fashion (Swamy et al., 2024), with the judge receiving the English reference response as privileged information.Thus, even when none of the model's responses are completely correct, the privileged pairwise judge can still tell which response is better.End-to-end, SP3F greatly improves base model performance, even outperforming fully post-trained models on multiple math and non-math tasks with less than 1/8 of the training data across the single-language, multilingual, and generalization to unseen language settings.Our key insight is that we can use English reference responses during both SFT and RL by framing both learning problems in terms of translation.In particular, we use reference responses as data for translation during SFT and as privileged information for the pairwise judge during downstream RL.More explicitly, our contribution is three-fold:
Institution: Carnegie Mellon University
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: hybrid
OpenAlex ID: W7166873423