Is a Document Educational or Just Wikipedia-Style? — Pitfalls of Classifier-Based Quality Filtering
Year: 2026
Type: conference-paper
Abstract: Classifier-based Quality Filtering has recently emerged as a fundamental technique in constructing pre-training corpora.The ability to deploy a single model that can replace or supplement a set of heuristics has proven effective across numerous Large Language Models.In this work, we expose a critical vulnerability in this approach by demonstrating how a straightforward Wikipedia-style reformatting operation can substantially alter a model's quality assessment and enable low-quality content to surpass filtering thresholds.Our analysis reveals that the FineWeb-Edu CQF model would reverse its filtering decision for approximately 7% of evaluated documents, thereby admitting content into the pre-training corpus that would otherwise have been excluded.1
Institution: Warsaw University of Technology
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
SDG: Quality education
Open Access status: hybrid
OpenAlex ID: W7166867478
