Year: 2026
Type: preprint
Abstract: The standard Transformer architecture has revolutionized sequence modeling, yet it is fundamentally limited by its quadratic computational complexity and infinite memory capacity requirement, which scales proportionally with the sequence length. This contradicts biological working memory, which compresses past information into finite capacity representations. We propose to address this bottleneck by introducing a Memory Component that integrates several dynamical systems implemented as linear reservoir units inspired by Reservoir Computing. Each unit possesses its own learned and adaptive dynamics, with the objective of selectively storing previous inputs in a compact internal state. The core intuition behind this multi-state-vector memory is that the standard attention mechanism applied across the temporal sequence of tokens can instead be applied across these parallel memory state-vectors. By doing so, the attention computation becomes constant per timestep, as the number of memory units remains fixed regardless of the sequence length. Furthermore, because the memory implementation relies strictly on linearized reservoirs, the model can be interpreted simultaneously as a Transformer (fully parallelizable across the sequence during training) and as a Recurrent Neural Network (operating with a fixed-size state during inference). By leveraging matrix diagonalization, an associative scan algorithm, and a custom Triton CUDA kernel, our model yields O(log T) training complexity over a sequence while fully exploiting high-performance GPUs. We additionally introduce an adaptive leak rate mechanism, granting each memory unit the ability to dynamically modulates its retention around its preferred timescale, effectively instantiating a Selective State Space Model (SSM). We confirm the architecture's scalability and modeling capabilities on the Shakespeare dataset (1M tokens) and the OpenWebText dataset (9B tokens).
Source: HAL
Authors: Bendi-Ouis, Yannis, Hinaut, Xavier
Institution: MNEMOSYNE: Mnemonic Synergy
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7143583446