When Models Train on Their Own Homework : Dose-Response of Model Collapse in a Word-Level Transformer and the 75% Human-Text Line
Year: 2026
Type: preprint
Abstract: When language models are trained on text written by earlier models, quality, diversity and factual reliability can decline across generations. This is called model collapse. This paper quantifies how quickly the erosion occurs in a small system, and identifies how much human written text each generation’s training data needs to keep the vocabulary and diversity close to the original. For this, A word-level nanoGPT-style transformer was trained from scratch on CPU for three generations. The training data was 13,920 tokens from the opening passages of Charles Dickens’s 1859 novel A Tale of Two Cities, and the result was evaluated against the validation data of 1,579 tokens from the following passages. Two arms were then trained recursively for the three generations. In the collapse arm, each generation trains only on the previous generation’s output. In the mitigated arm, each generation mixes fresh real text with a synthetic share s of that output. Two experiments cover nine values of s, five seeds each, 45 runs in all, and every result below is a mean with a 95% confidence interval. The collapse arm loses two-thirds of its rare words by generation 3 (retention fell from 0.652 to 0.208), and the vocabulary of generated text shrank by 59.6% (1,821 to 736 words). However, mixing in real data reduced the loss. At synthetic shares of 0.1 to 0.25, models kept 72-88% of their original rare words coverage, and no run produced a repetition loop. At s ≥ 0.3, 6 of 20 runs produced repetition loops. Vocabulary per set fell with synthetic share. Average metrics barely separated s = 0.25 from s = 0.3, but worst-case sets did, which is exactly why worst-case checking belongs in every collapse evaluation. Real data protects against collapse in proportion to its share. For this setup, keeping at least 75% human-written data per generation kept the model near the real-text floor. The thresholds should be read as indicative: one novel, one small model, five seeds.
Author: Saraf, Misha
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7233560960
Cites:
Cited by:
Related to:
