Where Induction Runs Out: Description-Length Difficulty and the Memorisation Gap in Integer-Sequence Benchmarks
Year: 2026
Type: preprint
Abstract: Integer sequences from the OEIS are increasingly used to benchmark mathematical reasoning in language models. We ask what such benchmarks actually measure, using an exactly computable reference learner: two-part minimum description length (MDL) over the class of P-recursive (holonomic) recurrences, evaluated on every prefix of a sequence as terms arrive. Three findings follow. First, MDL difficulty is a parameter count. The discovery point n_d, the first prefix length at which a symbolic hypothesis beats verbatim storage, is predicted almost exactly by a combinatorial identifiability bound on the selected operator's order and degree. It is invariant to term magnitude: scaling Fibonacci over twelve orders of magnitude leaves n_d unchanged, because a hypothesis must encode its own initial conditions and the magnitude cancels. Second, at scale the learner exhibits a regime our curated corpus could not produce even once: across 20,000 OEIS sequences, 89.98% of those that fit a recurrence on some prefix fit none at full length. We call this the wilderness - induction acquires a theory, loses it, and never recovers. Third, evaluating three language models on sequences stratified by these MDL regimes refuted our pre-registered hypothesis: models do not confabulate where MDL reports no theory, but hedge appropriately. Confident errors are inverted, concentrating on the easy stratum, where apparent competence tracks recognition of the sequence rather than induction of its rule. OEIS-derived benchmarks therefore substantially measure memorisation, and MDL supplies a cheap, contamination-free difficulty signal they currently lack. Code, data and all model responses are released at https://github.com/sabilashang/where-induction-runs-out --- Version notes (v1.1): Readability pass - first-use expansion of acronyms(OEIS and others), a notation table in Setup, and a worked Fibonacci encodingexample following the two-part MDL cost derivation. The compiled paper is now14 pages (was 13). CITATION.cff now correctly points at the software DOI(10.5281/zenodo.21830532); the README adds a matching software-DOI badge andseparates the paper and software citation links. Added a conditional buildflag supporting an anonymised variant for double-blind workshop submission(arXiv/Zenodo copy is unaffected). No results, numeric claims, or citationschanged from v1.0.0.
Author: Ganeshan, Sabilashan
Language: English
Subfield: Computational Theory and Mathematics
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7201864765
Cites:
Cited by:
Related to: