Work

Gradient flow for deep factored scalar linear attention learns multi-step gradient descent

Year: 2026
Type: preprint
Abstract: We study in-context learning of scalar linear regression by L untied linear-attention layers, each with a trainable key scalar k_i and query scalar q_i. For every depth L and prompt length n, population gradient flow on the 2L factors is bounded and converges to a critical point, every critical point that is not a global minimizer has a negative Hessian eigenvalue, and the flow reaches a global minimizer from all initializations outside a Lebesgue null set, for instance almost surely under Gaussian initialization. The trained network performs L gradient descent steps on the prompt's least-squares loss, with step sizes k_iq_i the reciprocals of the zeros of a rescaled Laguerre polynomial L_L^(n/2), and its risk is $2^LL!/\prod_{j=1}^{L}(n+2j)$. Initializations with k_i = -q_i in some layer converge to a critical point that is not a minimizer, so the null set is not empty.
Language: English
Open Access status: green
OpenAlex ID: W7233559841
Cites:
Cited by:
Related to: