Year: 2026
Type: conference-paper
Abstract: Mid-training has become critical for enhancing the knowledge and reasoning ability of large language models (LLMs), especially through the utilization of large-scale synthetic data.However, existing data synthesis methods often generate simplistic and homogeneous QA pairs, with limited scale and diversity.To address this, we propose BoostQA, a novel framework designed to synthesize large-scale, diverse, and high-quality QA data for midtraining.BoostQA introduces model probes during mid-training for the first time and implements STEM-focused multi-grade synthesis to boost data diversity as well as high-difficulty synthesis to alleviate difficulty degradation, followed by answer refinement to further improve quality.Extensive experiments by mid-training Llama-3 8B demonstrate that using only 20Btoken BoostQA data achieves a significant average improvement of 12.74% on MMLU and CMMLU over the pre-training baseline.After mid-training on 500B tokens, including 100Btoken BoostQA data, our model achieves SOTA average results across benchmarks among mainstream models of comparable size.BoostQA also demonstrates robust scalability, with performance consistently improving as model size, data volume, and initial FLOPs scale. 1 G L M 4 -9 B -B a s e L l a m a 3 .1 -8 B -B a s e Y i -1 .5 -9 B -B a s e Q w e n 2 .5 -7 B -B a s e G e m m a 2 -9 B -B a s e O u r s 0 20 40 60 80 MMLU 73.6 65.5 70.7 74.1 72.0 76.2 G L M 4 -9 B -B a s e L l a m a 3 .1 -8 B -B a s e Y i -1 .5 -9 B -B a s e Q w e n 2 .5 -7 B -B a s e G e m m a 2 -9 B -B a s e O u r s 0 20 40 60 80 CMMLU 75.0 53.1 75.0 82.8 59.1 80.
Authors: Xuemiao Zhang, Chengying Tu, Can Ren, Rongxiang Weng, Hongfei Yan, Jingang Wang, Xunliang Cai
Institutions: Peking University, Meituan (China)
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
SDG: Quality education
Open Access status: hybrid
OpenAlex ID: W4417098694