Token uniformity in residual self-attention at random initialization comes from the mean token
Year: 2026
Type: preprint
Abstract: We study residual self-attention networks X_(l+1) = X_l+α_l A_l X_l W_l at initialization, with residual scales α_l ≥ 0. The value matrices W_l are Gaussian. The attention matrices A_l are row-stochastic and may depend on the tokens and on any randomness independent of the value matrices, which covers softmax attention with query and key matrices independent of the value matrices. Let S_l be the squared Frobenius norm of X_l after the mean token is subtracted from every token. If the tokens have dimension d ≥ 2 and the input X_0 is deterministic with S_0>0, then log S_l is a submartingale for every choice of attention and of residual scales; this can fail for d = 1. Consequently, for a network of depth L with α_l = c/√L, the ratio S_L/‖X_L‖_F² stays bounded below in probability, uniformly in L and in the attention. At a fixed scale α it decays at most at the exponential rate log (1+α²n), where n is the number of tokens.
Author: Shmalo, Yitzchak
Institution: Hebrew University of Jerusalem
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7236150132
