Year: 2026
Type: conference-paper
Abstract: Adapting large language models (LLMs) to low-resource languages (LRLs) is constrained by the scarcity of task data and computational resources.Although Proxy Tuning offers a logit-level strategy for introducing scaling effects, it often fails in LRL settings because the large model's weak LRL competence might overwhelm the knowledge of specialized smaller models.We thus propose TRIMIX, a test-time logit fusion framework that dynamically balances capabilities from three different sources: LRL competence from a continually pretrained small model, task competence from high-resource language instruction tuning, and the scaling benefits of large models.It is data-and compute-efficient, requiring no LRL task annotations, and only continual pretraining on a small model.Experiments across four model families and eight LRLs show that TRIMIX consistently outperforms single-model baselines and Proxy Tuning.Our analysis reveals that prioritizing the small LRL-specialized model's logits is crucial for success, challenging the prevalent largemodel-dominant assumption.
Authors: Chen Zhang, Jiuheng Lin, Zhiyuan Liao, Yansong Feng
Institution: Peking University
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: hybrid
Award: G7590666161
OpenAlex ID: W7166864766