Kernel, Transport, Support: A Unified Prescription for Attention, State-Space and Delta-Rule Sequence Mixers
Year: 2026
Type: preprint
Abstract: Softmax attention, sliding-window attention, linear attention, state-space models and delta-rule recurrences such as Gated DeltaNet and Kimi Delta Attention are usually presented as separate families. This note writes them in one parameterization: a kernel (bilinear or exponential), a transport given by products of per-step transitions, and a set of kept tokens or memory slots, all driven by local queries, keys, values and gates. We separate exact re-realizations of the same input–output map from structural differences. Re-realizations are evaluation order, primal versus dual state, and time-dependent changes of state basis; the familiar "gauge" that turns complex diagonal recurrences into rotary embeddings is a special case. Structural differences are described by a signature: the kernel, the write rank, the type of the transitions and the type of support. A signature is a description, not an invariant, and separations between function classes are cited, not derived. A gauge depending only on position turns fixed transports into positional features. For data-dependent transports, a gauge relocates the transport into prefix-dependent features; it does not remove it. As a worked example we compile the known coordinate forms of the memory of sliding-window attention, a delay line of length W: shift register, ring buffer, and a Fourier form that is a single tied delta-rule step with unit-modulus complex gates. With real gates, such a step has Jordan chains of length at most three, which follows from known indefinite-inner-product arguments. We tabulate the layer composition of eleven released models and close with design heuristics, each stated with its evidence. The note proves no new theorem; it aims at a precise, attributed common language for these models.
Author: Pei, Yan Ru
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7226010210
