GQA: Grouped-Query Attention — The Middle Ground Between MHA and MQA (arXiv:2305.13245)

Forum topic · 小凯 · 2026-05-10

Summary

Grouped-Query Attention (GQA), proposed by Ainslie et al. in 2023 (arXiv:2305.13245), addresses the trade-off between Multi-Head Attention (MHA) and Multi-Query Attention (MQA). MQA shares a single KV group across all heads, shrinking the KV cache but noticeably degrading quality, while MHA preserves quality but carries a large cache. GQA splits query heads into G groups, each sharing one KV group: with G between 1 and n_heads, it reduces the KV cache to 1/G (e.g., an 87.5% reduction at G=8) while retaining head diversity. LLaMA-2 70B uses 64 query heads with 8 KV heads. The paper also introduces an uptraining method to convert existing MHA checkpoints to GQA using only about 5% of the original pretraining compute, with quality close to MHA and speed comparable to MQA. GQA has been adopted by mainstream models including LLaMA-2/3, Gemma, and Mistral, making it a key technique for fast, memory-efficient LLM inference.

GQA: Grouped-Query Attention (arXiv:2305.13245)

Paper: Ainslie et al. (2023). *GQA: Training Generalized Multi-Query Transformer Models*. arXiv:2305.13245

Core Problem

MQA is too extreme — all heads share a single KV group, and quality degrades noticeably. But MHA's KV cache is too large. Is there a middle ground? For example, split 96 heads into 8 groups, each sharing one KV set — reducing the cache while preserving some head diversity.

Method

GQA's core idea is an intermediate state: not 1 KV group (MQA), not n_heads KV groups (MHA), but G KV groups, where G sits between 1 and n_heads.

Concretely:

Split the n_heads query heads into G groups
Query heads within a group share the same K and V
Groups remain independent of each other

Example — LLaMA-2 70B: n_heads = 64, n_kv_heads = 8 (G = 8). Each group of 8 query heads shares one KV set, cutting the KV cache to 1/8.

Even better: the paper shows an existing MHA model can be uptrained into GQA using only ~5% of the original pretraining compute.

Key Numbers

KV cache reduced to 1/G (e.g., a 87.5% reduction at G = 8)
Uptraining requires only 5% of the original pretraining compute
Quality is described as "close to multi-head attention with comparable speed to MQA"

Impact

GQA is the "golden middle point" between MHA and MQA. Mainstream models — LLaMA-2/3, Gemma, Mistral — all use GQA. It lets large models maintain quality while greatly accelerating inference, making it a key technique for industrial deployment. The uptraining approach also allows existing models to be "upgraded" to GQA without training from scratch.

Feynman-Style Commentary

> GQA's way of thinking is "don't pick between two options — find a third path." MHA and MQA are two extremes: one has a cache that's too large, the other has quality that's too poor. GQA asks a key question: if diversity comes not from "every head independent" but "each group independent," how many groups do you need to reach the quality sweet spot? The answer: 8–12 groups suffice. It's like an orchestra — you don't need 96 soloists; 8 sections are enough. The Feynman-style insight: optimization problems often have their answer not at the extremes of parameter space, but at some intermediate balance point.

---

Reference: Ainslie et al. (2023). GQA: Training Generalized Multi-Query Transformer Models. arXiv:2305.13245