Year: 2025
Type: conference-paper
Abstract: We present Autonomous Data Selection (AutoDS), a method that leverages base language models themselves as zero-shot "generative classifiers" to automatically curate highquality mathematical texts.Unlike prior approaches that require human annotations or training a dedicated data filter, AutoDS relies solely on a model's logits to determine whether a given passage is mathematically informative and educational.By integrating Au-toDS into a continual pretraining pipeline, we substantially boost downstream performance on challenging math benchmarks (MATH, GSM8K, and BBH) while using far fewer tokens than previous methods.Empirically, our approach achieves roughly a twofold improvement in pretraining token efficiency over strong baselines, underscoring the potential of selfdirected data selection in enhancing mathematical reasoning.We release our curated AutoMathText dataset to facilitate future research in automated domain-specific data curation § .The AutoMathText dataset is available at https://huggingface.co/datasets/math- ai/AutoMathText.
Authors: Yifan Zhang, Yifan Luo, Yang Yuan, Andrew C Yao
Institutions: Tsinghua University, Shanghai Qi Zhi Institute
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
SDG: Quality education
Open Access status: gold
OpenAlex ID: W4412888688