Sillage: Surprise-Gated Amplitude Memory for Frozen Language Models
Year: 2026
Type: preprint
Abstract: We augment frozen language models with a fixed-size Hebbian associative memory that adapts at inference time with no gradients, no backpropagation, and no growth in storage. The memory is a single matrix (4.2 MB). Keys are sliding n-gram bindings of random token hypervectors, whose retrieval kernel is near-orthogonal except under exact repetition; values are token hypervectors whose stored coefficients are maintained as amplitudes -- square roots of accumulated mass -- by a local read-modify-write rule inspired by the classical square-root (Bhattacharyya) embedding of probability distributions; and every write is gated by the model's own token surprise -ln p_ LM , a three-factor plasticity rule whose modulator is free at inference. We call the system . On a 36k-token stream of novel technical text, the memory improves GPT-2's test negative log-likelihood by +0.486 +/- 0.005 nats (perplexity 31.2 -> 19.2; mean +/- SEM over 5 hypervector seeds), outperforming an unbounded kNN-LM (+0.281; paired block-bootstrap P=1.000) with 13x less memory , an exact n-gram cache (+0.318), a byte-matched kNN-LM (+0.174), and retrieval-augmented rescoring (+0.043). On a second, modern model (Qwen3-0.6B) the mechanism orderings replicate and statistically matches unbounded kNN-LM at one-sixteenth the memory; on a downstream content-token recall task, the augmented models complete recurring technical terms at up to 2.1x the frozen model's accuracy after a single reading pass. Ablations attribute the gain: amplitude encoding contributes +0.28 nats over count encoding (the largest single factor; P=1.000 in every seed), surprise gating quadruples the gain of uniform writes, and a multi-scale key doubles gains on low-repetition text. At 500k tokens the fixed matrix saturates (0.5 writes per parameter); leaky decay recovers 2.3x of the gain and a 4x larger matrix recovers 3.4x, quantifying a capacity-sizing law, while unbounded kNN-LM (grown to 770 MB) dominates on long narrative -- an honestly mapped regime boundary. Two evaluation artifacts (a corpus-formatting pathology producing +1.35 nats of phantom gains, caught by a shuffled-retrieval control; an off-by-one in the RAG baseline) are documented so others avoid them. Code and data scripts are released.
Author: sghairi, abderrahmane
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7204117523
Cites:
Cited by:
Related to: