Repetition over Diversity: High-Signal Data Filtering for Sample-Efficient German Language Modeling
Year: 2026
Type: conference-paper
Abstract: Recent research has shown that filtering massive English web corpora into high-quality subsets significantly improves training efficiency.However, for high-resource non-English languages like German, French, or Japanese, aggressive filtering creates a strategic dilemma: should practitioners prioritize diversity by training once on large amounts of lightly filtered web data, or prioritize quality by strictly filtering for a high-quality core and repeating it over multiple epochs?We investigate this trade-off for German by applying hierarchical quality filters to 500M web documents, comparing multi-epoch training on filtered subsets against single-pass training on a diverse corpus.Our experiments across multiple model scales and token budgets show that repeating high-quality data consistently outperforms single-pass training on larger, less filtered sets.Notably, the performance gap persists even after 7 epochs.Our findings suggest that for non-English LLMs, semantic concentration through quality filtering offers a more viable path to efficient language modeling than simply maximizing unique data volume.We release our German language models (called BOLDT), as well as our cleaned evaluation benchmarks to the research community.Our experiments indicate that they achieve state-of-the-art results despite training on 10-360x fewer tokens than comparable models.Contributions.This work makes the following contributions:• We demonstrate that repeated training on highquality subsets outperforms maximizing unique web document coverage under a fixed pretraining budget in German language modeling.We specifically analyze the distinct impacts of our three semantic tiers: Coherence, Information Value, and Educational Quality.• We show that multi-epoch pre-training on filtered data positively impacts instruction-tuning behavior, leading to higher correctness in assistant tasks.• We identify and correct significant noise in existing German translations of ARC-Challenge, Hellaswag, Lambada, and OpenBookQA benchmarks.We release these updated benchmarks to ensure more reliable assessment of LLM performance in the German NLP community.• We release a series of German Small Language Models (SLMs) up to 1B parameters.Despite being trained on an order of magnitude fewer tokens than competitive baselines, our models achieve state-of-the-art results for their size, underscoring the importance of high-quality data selection in resource-constrained settings.
Authors: Ansar Aynetdinov, Patrick Haller, Alan Akbik
Institution: Humboldt-Universität zu Berlin
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: hybrid
OpenAlex ID: W7159602988