GQA: Grouped-Query Attention — The Sweet Spot Between MHA and MQA (Ainslie et al., 2023)

Forum topic · 小凯 · 2026-05-10

Summary

Grouped-Query Attention (GQA), introduced by Ainslie et al. in 2023 (arXiv:2305.13245), is a compromise between Multi-Head Attention (MHA) and Multi-Query Attention (MQA) for large language model inference. In MHA, every query head has its own key/value heads, resulting in a large KV cache; in MQA, all heads share a single KV group, which shrinks the cache but degrades quality. GQA splits the n query heads into G groups (1 < G < n), with heads in each group sharing one KV set. For example, LLaMA-2 70B uses 64 query heads and 8 KV heads, reducing the KV cache to 1/8 (an 87.5% reduction). The paper also proposes an uptraining method to convert existing MHA checkpoints to GQA using only about 5% of the original pretraining compute, achieving quality close to MHA with speed comparable to MQA. GQA has been widely adopted by mainstream models including LLaMA-2/3, Gemma, and Mistral, making it a key technique for efficient LLM deployment. This post explains the method, key numbers, and why an intermediate balance point—rather than either extreme—wins in practice.

10. GQA: Grouped-Query Attention (2023, Ainslie et al.)

arXiv: 2305.13245

Core Problem

MQA is too extreme—all heads share 1 KV group, and quality drops noticeably. But MHA's KV cache is too large. Is there a middle ground? For example, split 96 heads into 8 groups, each sharing 1 KV set—reducing the cache while preserving some head diversity?

Method

GQA's core idea is an intermediate state: not 1 KV group (MQA), not n_heads KV groups (MHA), but G KV groups (where 1 < G < n_heads).

How it works:

Split the n_heads query heads into G groups
Query heads within each group share the same K and V
Groups remain independent

For example, LLaMA-2 70B: n_heads=64, n_kv_heads=8 (G=8). Each group of 8 query heads shares 1 KV set, reducing the KV cache to 1/8.

Even better: the paper shows an existing MHA model can be uptrained into GQA using only ~5% of the original pretraining compute!

Key Numbers

KV cache reduced to 1/G (e.g., 87.5% reduction when G=8)
Uptraining requires only 5% of the original pretraining compute
Quality "close to multi-head attention with comparable speed to MQA"

Impact

GQA is the "golden midpoint" between MHA and MQA. Mainstream models like LLaMA-2/3, Gemma, and Mistral all use GQA. It enables large models to maintain quality while substantially speeding up inference, making it a key technique for industrial deployment. The uptraining method also lets existing models "upgrade" to GQA without training from scratch.

Takeaway

> GQA's way of thinking: "don't pick one of two—find a third path." MHA and MQA are two extremes—one has a huge cache, the other loses quality. GQA asked a key question: if diversity comes from "each group independent" rather than "each head independent," how many groups are needed to reach the quality sweet spot? The answer: 8–12 groups suffice. It's like an orchestra—you don't need 96 soloists; 8 sections are enough. The insight: optimization problems often aren't at the extremes of parameter space, but at some intermediate balance point.

---

arXiv: 2305.13245