The Geometric and Algebraic Structure of Transformer Attention
Year: 2026
Type: preprint
Abstract: This paper develops a unified geometric and algebraic analysis of the attention mechanism in Transformer architectures. The central observation is that, given a token embedding matrix $X\in R^{N\times d}$, every similarity matrix producible by a single attention head lies inside a nested hierarchy $\mathcal{V}(d_k)(\mathbf{X}) \subset \mathcal{V} (\mathbf{X}) \subset \mathbb{R}^{N \times N}$ is a vector subspace of dimension at most $r^2$ (with $r=rk(X)$) and $V^{(d_k)}(X)$ is its intersection with the rank-$d_k$ determinantal variety. From this structure we derive results of three kinds. \emph{Principal contributions}: (i) a matching necessary-and-sufficient \emph{discriminability theorem} showing that the minimum number of attention heads achievable by some choice of query projections is exactly $\lceil d/d_k\rceil$, providing an \emph{a priori} structural justification for the standard design $h\cdot d_k=d$ that complements the capacity-based account and addresses the gap; (ii) a tighter effective-rank version $\lceil r/d_k\rceil$ when $rk(X)=r
Author: Sentoni, Guillermo Blas
Institution: Universidad Nacional de La Matanza
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7162102497
Cites:
Cited by:
Related to: