Grouped-Query Attention vs. Standard Multi-Head Attention: A Controlled Comparison at Small Scale
Year: 2026
Type: preprint
Abstract: We present a controlled empirical comparison between grouped-query attention (GQA) and standard multi-head attention (MHA) in a small (~13.6M parameter) GPT-style decoder-only Transformer, trained from scratch on the TinyStories dataset. Holding architecture, random seed, data, and training budget (30,000 steps) fixed, and varying only the number of key/value heads (2 for GQA vs. 4 for MHA, with 4 query heads throughout), we find the two configurations reach nearly identical validation loss (2.2500 for GQA vs. 2.2451 for MHA; perplexity 9.49 vs. 9.44) while GQA uses exactly half the key/value projection parameters (65,536 vs. 131,072). This small, controlled result is consistent with the finding reported for GQA at much larger scale: the technique gives a real parameter and memory saving at negligible cost to model quality, and this cost remains negligible even at a scale three-plus orders of magnitude smaller than where GQA is typically studied.
Author: Kapil
Institution: University of Petroleum and Energy Studies
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7203697017
Cites:
Cited by:
Related to: