Gated Causal Average: A Query-Free, Linear-Time Token Mixer for Language Models
Year: 2026
Type: preprint
Abstract: Self-attention costs O(T²) in sequence length. We introduce the Gated Causal Average (GCA), a causal token-mixing operator that computes, separately in every channel, a content-weighted running average of the prefix: each token i receives a positive weight vector wᵢ = softplus(W yᵢ + b), and the output at position t is the sum of wᵢ ⊙ yᵢ over i ≤ t divided by the sum of wᵢ over i ≤ t. GCA has no query, no value projection and no pairwise score; it has D independent denominators, runs in O(T) time with prefix sums, and needs no custom kernel. With its gate initialised to zero it computes exactly the unweighted causal mean, so a gated model starts from the ungated baseline. We pair GCA with a causal depthwise convolution in a single mixer and train attention-free language models (84.6–86.7M parameters) on BabyLM strict-small (10M words). On BLiMP, the bottleneck-gated GCA-V2 scores 67.59 and the single-layer gate GCA-V1 67.2, against 66.52 for the ungated causal mean (single seed; two training configurations, reported separately). The nearest prior work is the Attention-Free Transformer; we state the three differences exactly so that the novelty claim can be checked.
Author: Abass, Abdalhay
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7225596822