Work

Hierarchical Data Selection via Manifold Coverage and Sparse Feature Coverage in LLM Post-training

Year: 2026

Type: preprint

Abstract: As supervised fine-tuning data continues to scale, selecting high-value subsets from large candidate pools is crucial for reducing training cost and improving model performance. Existing methods often measure diversity directly in the original embedding space, where geometric metrics entangle dominant semantic directions, fine-grained supervision differences, and local noise. We address this limitation by formulating data selection as a coarse-to-fine hierarchical coverage problem and propose MASS. MASS learns low-dimensional principal manifold coordinates with a dense autoencoder for coarse semantic grouping, and then performs quality-aware sparse feature coverage within each group using a TopK sparse autoencoder. Experiments on Vision Flan and LLaVA-CoT show that MASS consistently outperforms strong data selection baselines across multiple budgets, and in several settings matches or surpasses full data training with only a small subset of data.

Source: arXiv (Cornell University)

DOI: 10.48550/arxiv.2608.16927

Authors: Sun, Peng, Yang, Yi, Zhang, Antong, Li, Chunxiao, Wang, Yanbo, Liu, Dianbo, chen, xin, Yu, Kai, Chen, Lu, Fu, Tianfan

Institutions: Nanjing University, Brown University, Fudan University, North University of China, National University of Singapore, Shanghai Jiao Tong University

Topic: Topic Modeling

Subfield: Artificial Intelligence

Field: Computer Science

Domain: Physical Sciences

Open Access status: green

OpenAlex ID: W7203747544

Cites:

Cited by:

Related to: