Work

Your Query is My Key: Halving Attention Parameters via Cross-Head Cyclic Rings

Year: 2026
Type: article
Abstract: Query and key projections in self-attention enter the score through their product, which creates a continuous redundancy in their parameterisation. Tying the projections removes this freedom but restricts the score matrix. This paper derives the corresponding gradient identity, describes the symmetry and sign constraints of direct tying, and introduces a cross-head cyclic construction that uses one projection per head and forms scores from neighbouring projections in a ring. The query-key parameter count is reduced by half. Experiments cover associative recall, character-level language modelling, memory use, and small-scale expressiveness tests. In several tested settings, the cyclic model reaches high task accuracy while using half the query-key parameters. The measurements do not establish a general advantage, but they show that the cyclic restriction can retain strong performance at the tested scale
Language: English
Open Access status: green
OpenAlex ID: W7236599873
FWCI:
Cites:
Cited by:
Related to: 10