GQA: Grouped-Query Attention (2023, Ainslie et al.) — A Middle Ground Between MHA and MQA

Forum topic · 小凯 · 2026-05-10

Summary

Grouped-Query Attention (GQA), introduced by Ainslie et al. in 2023 (arXiv: 2305.13245), addresses the trade-off between Multi-Head Attention (MHA) and Multi-Query Attention (MQA). While MQA reduces the KV cache by having all query heads share one KV group, it noticeably degrades quality; MHA preserves quality but carries a large KV cache. GQA splits query heads into G groups, each sharing one set of keys and values, reducing the KV cache to 1/G (e.g., 87.5% reduction at G=8) while retaining head diversity. For example, LLaMA-2 70B uses 64 query heads and 8 KV heads. The paper also proposes an uptraining method to convert existing MHA checkpoints into GQA using only about 5% of the original pretraining compute, with quality close to MHA and speed comparable to MQA. GQA has been adopted by major models including LLaMA-2/3, Gemma, and Mistral, making it a key technique for efficient large-model inference and industrial deployment.

Background: The Problem

Core issue: MQA (Multi-Query Attention) is too extreme — all heads share a single KV group, causing noticeable quality degradation. But MHA (Multi-Head Attention) has a KV cache that is too large. Is there a middle ground? For example, split 96 heads into 8 groups, with each group sharing one set of KV — reducing cache while preserving some head diversity?

Method

GQA's core idea is an intermediate state: not 1 KV group (MQA), nor n_heads KV groups (MHA), but G KV groups (where 1 < G < n_heads).

Concretely:

Split the n_heads query heads into G groups
Query heads within each group share the same K and V
Groups remain independent of each other

Example: LLaMA-2 70B uses n_heads=64 and n_kv_heads=8 (G=8). Each group of 8 query heads shares one KV set, shrinking the KV cache to 1/8.

Even better: the paper shows an existing MHA model can be uptrained into GQA using only ~5% of the original pretraining compute.

Key Numbers

KV cache reduced to 1/G (e.g., 87.5% reduction at G=8)
Uptraining requires only 5% of original pretraining compute
Quality described as "close to multi-head attention with comparable speed to MQA"

Impact

GQA is the "golden midpoint" between MHA and MQA. Mainstream models including LLaMA-2/3, Gemma, and Mistral all adopt GQA. It enables large models to maintain quality while significantly accelerating inference, making it a key technology for industrial deployment. The uptraining method also lets existing models be "upgraded" to GQA without training from scratch.

Feynman-Style Takeaway

> GQA's mindset is "don't choose between two extremes — find a third way." MHA and MQA are two extremes: one has too large a cache, the other too poor quality. GQA asks a key question: if diversity means "each group independent" rather than "each head independent," how many groups reach the quality sweet spot? The answer: 8–12 groups suffice. It's like an orchestra — you don't need 96 soloists; 8 sections are enough. The Feynman-style lesson: optimization often lies not at the extremes of parameter space, but at some intermediate balance point.

---

arXiv: 2305.13245