Directional Consensus Mapping: Zero-Shot Optimization Trajectory Alignment for Pre-Training Data Selection
Year: 2026
Type: preprint
Abstract: Selecting high-quality, task-aligned pre-training data is critical for training foundation Large Language Models (LLMs). Current automated data curation frameworks rely on heuristic proxies—specifically token-level unigram KL-divergence (e.g., DSIR) or frozen representation-space distance metrics (e.g., SemDeDup). We demonstrate that these geometric proxies fail on multimodal and structurally heterogeneous distributions because distance in activation space is uncoupled from the optimization loss manifold. In this technical report, we propose Directional Consensus Mapping (DCM): a first-principles optimization framework that determines pre-training data suitability directly within the parameter tangent space at initialization (W_0). By projecting per-sample gradients onto the unit hypersphere ("one sample, one vote") and computing the directional consensus vector field across network layers, DCM isolates the pure angular alignment (cosine similarity) between candidate updates and the target benchmark descent trajectory. In preliminary empirical benchmarks on an untrained 1.2-billion-parameter transformer (Llama-3.2-1B, probing 973,144,064 reasoning parameters) across six diverse open-source corpora using only 80 samples per domain, DCM achieves a Spearman rank correlation of rho = +0.7143 against ground-truth downstream loss. Crucially, DCM naturally resolves catastrophic formatting magnitude bias, accurately isolating mathematical reasoning signals (cos theta ~ 0.39–0.43) from unaligned dialogue (cos theta ~ 0.19) and syntax-heavy code (cos theta ~ 0.06).
Author: JAIN, SHASHANK
Institution: University of Campania "Luigi Vanvitelli"
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7225379944