Year: 2025
Type: conference-paper
Abstract: It is well-known that a diverse corpus is critical for training large language models, which are typically constructed from a mixture of various domains.In general, previous efforts resort to either sampling training data from different domains with static proportions or dynamically adjusting these proportions during training to optimise pretraining performance.However, few methods addressed the complexity of domain-adaptive continual pre-training.To fill this gap, we propose Velocitune, a novel framework that dynamically assesses learning velocity and adjusts data proportions accordingly, favouring slower learning domains while de-emphasising faster learning ones, which is guided by a scaling law to estimate the desired learning goal for each domain with a less associated cost.To evaluate the effectiveness of Velocitune, we conduct experiments on a dataset focused on reasoning tasks with CodeLlama, as well as on a corpus of system commands using Llama3 and Mistral.Velocitune achieves performance gains in both math and code reasoning tasks and command-line generation benchmarks.Further analysis reveals that key factors driving the effectiveness of Velocitune include target estimation and data ordering.* The first two authors contribute equally.
Institutions: University of Manchester, Tsinghua University
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: hybrid
OpenAlex ID: W4412945071