Year: 2026
Type: conference-paper
Abstract: We investigate self-training of large language models (LLMs), in which models are recursively trained on their own generated text.Prior studies have reported conflicting outcomes in this setting: some find evidence of performance gains (i.e., self-improvement), while others observe performance degradation (i.e., model collapse).To clarify this discrepancy, we use the OLMo-2 models as non-toy LLMs and perform multiple rounds of continual pre-training using self-generated text with varying data synthesis and filtering strategies.Our experiments show that naive self-training does not improve either perplexity on the original pre-training corpus nor downstream task performance, regardless of model size.These results suggest that model collapse observed in naive self-training is inherent to the training procedure, while selfimprovement likely owes its success not to the model's autonomous refinement but to humandesigned, strategic synthetic pipelines that inject external intelligence.
Institutions: Waseda University, National Institute of Informatics
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: hybrid
OpenAlex ID: W7140092493