GQA: Grouped-Query Attention — Trading KV Cache Size Against Quality (arXiv 2305.13245)

Forum topic · 小凯 · 2026-05-10

Summary

Grouped-Query Attention (GQA), proposed by Ainslie et al. in 2023 (arXiv 2305.13245), introduces a middle ground between Multi-Head Attention (MHA) and Multi-Query Attention (MQA). MQA shares a single set of key/value (KV) heads across all query heads, shrinking the KV cache but noticeably degrading quality; MHA keeps a full KV head per query head, preserving quality but inflating the cache. GQA groups query heads into G groups, each sharing one KV head set. For example, LLaMA-2 70B uses 64 query heads and 8 KV heads, reducing the KV cache to 1/8 (an 87.5% reduction). The paper also shows an existing MHA model can be converted to GQA via 'uptraining' using only about 5% of the original pretraining compute. Quality is reported as close to MHA with speed comparable to MQA, with roughly 8–12 groups often sufficient. GQA has been adopted by LLaMA-2/3, Gemma, and Mistral, making it a key technique for efficient large-model inference.

10. GQA: Grouped-Query Attention (2023, Ainslie et al.)

arXiv: 2305.13245

Core Problem

MQA is too extreme: all query heads share 1 set of KV heads, and quality degrades noticeably. But MHA's KV cache is too large. Is there a middle ground — e.g., split 96 heads into 8 groups, each sharing 1 set of KV — reducing cache while retaining some head diversity?

Method

GQA's core idea is an intermediate state: not 1 KV set (MQA), not n_heads KV sets (MHA), but G KV sets (where 1 < G < n_heads).

Concretely:

Split n_heads query heads into G groups
Query heads within each group share the same K and V
Groups remain independent of each other

Example — LLaMA-2 70B: n_heads = 64, n_kv_heads = 8 (G = 8). Each group of 8 query heads shares 1 set of KV. The KV cache shrinks to 1/8.

Even better: the paper shows an existing MHA model can be uptrained into GQA — requiring only ~5% of the original pretraining compute.

Key Numbers

KV cache reduced to 1/G (e.g., G = 8 gives an 87.5% reduction)
Uptraining needs only 5% of the original pretraining compute
Quality described as "close to multi-head attention with comparable speed to MQA"

Impact

GQA is the "golden midpoint" between MHA and MQA. Mainstream models including LLaMA-2/3, Gemma, and Mistral all use GQA. It lets large models maintain quality while substantially speeding up inference, making it a key technology for industrial deployment. The uptraining method also lets existing models be "upgraded" to GQA without training from scratch.

Feynman-style Commentary

> GQA's way of thinking is: "don't pick one of two extremes — find a third path." MHA and MQA are two extremes — one has too large a cache, the other too poor quality. GQA asks a key question: if diversity means "independent per group" rather than "independent per head," how many groups are needed to hit the quality sweet spot? The answer: about 8–12 groups suffice. It's like an orchestra — you don't need 96 soloists; 8 sections are enough. The Feynman-style insight: optimization often lands not at the extremes of parameter space, but at some intermediate balance point.