Work

Relation Summaries for Dynamic Attention Head Routing: A Matched-Budget Study of Parity and Key-Value Retrieval

Year: 2026
Type: preprint
Abstract: This study tests whether information from attention heads within a Transformer layer helps a router decide how many additional heads to activate at a matched average head budget. The four-layer causal Transformer uses two always-on heads. The router can use their outputs and attention-weighted summaries of pairwise relations carried across layers. On the four-clue parity task, switching from fixed four-head routing to variable head counts raises mean test accuracy from 82.83% to 91.85%, with 3.988 active heads per query position on average. Removing the relation summary lowers accuracy in all three paired training seeds. On the key-value retrieval task, all three configurations use variable head counts and average approximately four active heads per query position. Across eight paired seeds, mean four-key exact-match accuracy is 89.02% with hidden states alone, 76.32% after adding the always-on head outputs, and 89.00% after also adding the relation summary. The additional routing inputs do not yield a consistent improvement over hidden-state routing on this task. Both tasks are synthetic and use a 128-token context. The implementation computes all candidate heads before applying hard masks, so the budget measures the number of active heads. Runtime and memory gains remain unmeasured.
Language: English
Open Access status: green
OpenAlex ID: W7232156302
Cites:
Cited by:
Related to: 10