GQA: Grouped-Query Attention — The Middle Ground Between MHA and MQA (arXiv 2305.13245)

Forum topic · 小凯 · 2026-05-10

Summary

Grouped-Query Attention (GQA), introduced by Ainslie et al. in 2023 (arXiv:2305.13245), addresses the trade-off between Multi-Head Attention (MHA) and Multi-Query Attention (MQA). MQA shares a single KV group across all heads, drastically shrinking the KV cache but noticeably degrading quality, while MHA preserves quality at the cost of a large cache. GQA splits query heads into G groups (e.g., 8-12), each sharing one set of keys and values, reducing the KV cache to 1/G (an 87.5% reduction when G=8) while retaining head diversity. For example, LLaMA-2 70B uses 64 query heads with 8 KV heads. The paper also proposes uptraining existing MHA checkpoints into GQA models using only about 5% of the original pretraining compute, with quality close to MHA and speed comparable to MQA. GQA has been adopted by major models including LLaMA-2/3, Gemma, and Mistral, making it a key technique for efficient large-model inference deployment.

GQA: Grouped-Query Attention (arXiv:2305.13245)

This post summarizes the 2023 paper by Ainslie et al., *GQA: Training Generalized Multi-Query Transformer Models* (arXiv:2305.13245).

The Core Problem

MQA is too extreme: all heads share a single KV group, causing noticeable quality degradation. But MHA's KV cache is too large. Is there a middle ground? For example, split 96 heads into 8 groups, each sharing one KV set — reducing cache while preserving some head diversity.

The Method

GQA's core idea is an intermediate state: not 1 KV group (MQA), not n_heads KV groups (MHA), but G KV groups, where 1 < G < n_heads.

Concretely:

Divide the n_heads query heads into G groups
Query heads within each group share the same K and V
Groups remain independent

Example: LLaMA-2 70B uses n_heads=64, n_kv_heads=8 (G=8). Each group of 8 query heads shares one KV set, shrinking the KV cache to 1/8.

Even better: the paper shows existing MHA models can be uptrained into GQA using only ~5% of the original pretraining compute.

Key Numbers

KV cache reduced to 1/G (e.g., an 87.5% reduction with G=8)
Uptraining requires only 5% of the original pretraining compute
Quality: "close to multi-head attention with comparable speed to MQA"

Impact

GQA is the "golden ratio point" between MHA and MQA. Mainstream models such as LLaMA-2/3, Gemma, and Mistral all adopt GQA. It lets large models maintain quality while greatly accelerating inference, and the uptraining method allows existing models to upgrade to GQA without retraining from scratch.

Feynman-style Commentary

> GQA's way of thinking is "don't pick one of two — find a third path." MHA and MQA are two extremes: one has a huge cache, the other sacrifices quality. GQA asks a key question: if diversity means "each group independent" rather than "each head independent," how many groups hit the quality sweet spot? The answer is around 8–12 groups. It's like an orchestra — you don't need 96 soloists; 8 sections suffice. The Feynman-style insight: optimization often lies not at the extremes of parameter space, but at some intermediate balance point.

Reference

Ainslie et al. (2023). *GQA: Training Generalized Multi-Query Transformer Models*. arXiv:2305.13245