MQA: Multi-Query Attention (2019, Shazeer et al.) — Cutting KV Cache by Sharing Keys and Values

Forum topic · 小凯 · 2026-05-10

Summary

Multi-Query Attention (MQA), introduced by Noam Shazeer et al. in arXiv:1911.02150, targets the real inference bottleneck of Transformers: memory bandwidth, not computation. In multi-head attention (MHA), each query head has its own Key and Value tensors, so the KV cache scales with the number of heads and can reach gigabytes at 96 heads, 128-dim heads, and 4K context. MQA makes all query heads share a single K and V head, shrinking the KV cache by a factor of n_heads while keeping the query projections unchanged. This drastically reduces memory traffic during autoregressive decoding, enabling much faster token generation with only minor quality degradation. Although the shared K/V loses head diversity and MQA itself was not widely adopted directly, it established the core principle that KV cache size — not attention compute — dominates decoding cost, paving the way for follow-ups like GQA, MLA, and SWA.

MQA: Multi-Query Attention (2019, Shazeer et al.)

Paper: arXiv: 1911.02150

The core problem: where is the Transformer inference bottleneck?

Not in computation (the forward pass is fast), but in memory bandwidth — every generated token requires loading the large Key and Value tensors from GPU memory into the compute units. In multi-head attention (MHA), each head has its own K and V, so the KV cache size = n_heads × d_head × seq_len. With 96 heads, 128-dim heads, and 4K context, this cache reaches the gigabyte scale. How can it be cut without seriously hurting quality?

The method: all query heads share one K and V

MQA's solution is radical: all query heads share a single set of Keys and Values.

In MHA:

Q: [batch, n_heads, seq_len, d_head]
K: [batch, n_heads, seq_len, d_head] ← n_heads copies
V: [batch, n_heads, seq_len, d_head] ← n_heads copies

In MQA:

Q: [batch, n_heads, seq_len, d_head] ← unchanged
K: [batch, 1, seq_len, d_head] ← only 1 copy
V: [batch, 1, seq_len, d_head] ← only 1 copy

The KV cache drops from n_heads copies to 1, drastically reducing memory bandwidth requirements and greatly speeding up decoding.

The cost: all heads see the same "memory," losing the ability of different heads to attend to different subspaces — quality degrades.

Key numbers

"Much faster to decode"
"Only minor quality degradation from the baseline"
Author: Noam Shazeer (one of the Transformer authors, later co-founder of Character.AI)

Impact

MQA was the first step in attention "slimming." It proved an important principle: the KV cache is the inference bottleneck, not computation. All subsequent attention optimizations (GQA, MLA, SWA) revolve around reducing the KV cache. However, MQA's quality drop meant it was not directly adopted by mainstream models — it functions more as a thought experiment, demonstrating that slimming is possible, and also that slimming too much hurts.

Feynman-style commentary

> MQA's real value is teaching you to identify the true bottleneck. Most people think Transformer inference is slow because attention computation is heavy — no, the slowness is memory bandwidth. The attention matrix computation is O(n²), but each token only computes one row, so it's actually fast. What's slow is moving the KV cache from GPU memory to the compute units. MQA isn't "better attention" — it's "deleting the parts of attention that don't need to be duplicated." It's like moving house: not throwing out furniture, but merging 96 identical chairs into 1.

---

arXiv: 1911.02150