Work

Hilberg's law as learning a world state: implications and limits for text compression

Year: 2026
Type: article
Abstract: Hilberg's hypothesis states that block entropy of text grows as hn + An^beta. We connect maximal entropy random walk (Parry) sequence ensembles, ANS-like equipartition of surprisal, Santa Fe processes with an HCR-motivated Gaussian world model, and LLM-based entropy measurements into a picture of text as progressive revelation of a world state. Calibration on Santa Fe processes shows that PPMd compression slopes barely respond to the true Hilberg exponent, while a word-level Pitman-Yor coder recovers it but measures lexicon occupancy. A permutation-calibrated lexical index detects word-order effects in anthologies. For compression, a self-contained coder cannot remove a genuine An^beta source term; explicit word and recency models reduce predictive log loss by 1.5-3.4% over a context-mixing base, and shared dictionaries pay off mainly for small independently coded documents. Supplementary files contain all code and results.
Language: English
Open Access status: green
OpenAlex ID: W7225885490
FWCI:
Cites:
Cited by:
Related to: 10