Sparse Long Memory and Transition Bottlenecks Shape Positional Recall in Mamba Language Models
Year: 2026
Type: conference-paper
Abstract: Mamba language models process sequences recurrently, but recurrent processing does not imply that all tokens are retained equally.We study this gap using structured recall tasks that expose primacy, recency, and lost-in-themiddle behavior.Our results suggest that Mamba uses two functional memory regimes: a short-lived state that supports recency, and a sparse long-lived channel subspace that preserves selected early inputs.Targeted interventions on this long-memory subspace selectively degrade early recall and transfer to long-context QA and summarization, indicating that the same channels support real longcontext use.We then ask why middle items are forgotten despite being processed sequentially.Our results support a short-to-long transition bottleneck account: middle items are no longer recent enough for short memory, but are often not promoted into the sparse long-lived subspace.Repeated input structure strengthens this bottleneck by changing the input-dependent discretization dynamics.These findings provide a mechanistic account of Mamba's U-shaped recall curve as the interaction between long memory, short memory, and a fragile transition between them.
Institutions: RIKEN, Tohoku University
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: bronze
OpenAlex ID: W7224445784