Get started
Work

Write Where Short Memory Cannot Reach: Distance-Invariant Fact Recall in Fixed-State Language Models

Year: 2026
Type: preprint
Abstract: Fixed-state recurrent language models such as Gated DeltaNet and Mamba promise constant memory at any context length, yet in practice they recall a fact a few thousand tokens later and then lose it. We remove both ways a delta-rule memory forgets, decay and erasure by later writes, on archival heads whose decay is pinned to one and whose writes must pass a hard threshold. On synthetic streams, 2.2M-parameter models trained on 256-token sequences recall up to 32 facts after 1M tokens in 9/9 seeds. For language models this is necessary but not sufficient: with truncated backpropagation, if a bounded-memory layer can answer the in-segment training queries, recall is routed through it and ends at its horizon. Making those queries unanswerable by bounded memory (the routing principle) yields 117M-parameter models whose recall of facts in FineWeb-Edu text is flat from 16K to 1M tokens. The same principle grafts the memory onto frozen pretrained models as small adapters. With a write-threshold curriculum, an LFM2-350M graft recalls 0.90 of facts read 1M tokens earlier on fresh streams in 0.84 GB of GPU memory, and a Qwen3.5-4B graft also 0.90; the level varies across training seeds (0.575-0.95). Short single-GPU runs transfer to LFM2.5-8B-A1B (0.35-0.375 at 1M) and Gemma 4 E2B (flat to 1M at 0.40-0.575). On BABILong, a short adapter fine-tune of the 350M graft gives 0.50-0.61 from 0K to 1M tokens, two to three times a retrieval baseline from 4K up. We document four training-signal traps and release code, adapters, a live demo and all evaluation logs. Live demo: huggingface.co/spaces/01TEK/KH-1-live (upload any book, read it into a 1.9 MB memory, then ask) · Video: youtu.be/mCGc3_Mh0HI · Code and every evaluation log: github.com/01tek/KH (tag v2.3) · Models: huggingface.co/01TEK Compute: Nebius, through the NVIDIA Inception program.
DOI: 10.5281/zenodo.23447664
Institution: MediaTek (China)
Language: English
Open Access status: green
OpenAlex ID: W7222871079
Cites:
Cited by:
Related to: