MQA: Multi-Query Attention (Shazeer et al., 2019) — Shrinking the KV Cache to One Head

Forum topic · 小凯 · 2026-05-10

Summary

Multi-Query Attention (MQA), introduced by Noam Shazeer in arXiv:1911.02150 (2019), addresses the true inference bottleneck of Transformers: memory bandwidth, not computation. During autoregressive decoding, the Key and Value tensors must be repeatedly loaded from GPU memory, and under standard multi-head attention (MHA) the KV cache scales as n_heads × d_head × seq_len — reaching gigabytes for models with 96 heads and 4K context. MQA's radical solution: all query heads share a single K and V projection, reducing the KV cache from n_heads copies to one while keeping queries unchanged. This drastically lowers memory bandwidth needs and greatly speeds up decoding, with only minor quality degradation, since heads lose the ability to attend to different subspaces. The post explains the tensor shapes in MHA vs MQA, quantifies the cache reduction, and situates MQA historically: it established the principle that KV cache (not FLOPs) dominates inference cost, inspiring later methods like GQA, MLA, and SWA. MQA itself was too lossy for wide adoption but served as the proof of concept for attention slimming.

9. MQA: Multi-Query Attention (2019, Shazeer et al.)

arXiv: 1911.02150

The core problem

Where is the inference bottleneck of a Transformer? Not in computation (the forward pass is fast), but in memory bandwidth — every generated token requires loading the huge Key and Value tensors from GPU memory into the compute units. In multi-head attention (MHA), each head has its own K and V, so the KV cache size = n_heads × d_head × seq_len. For a model with 96 heads, 128-dim heads, and 4K context, this cache is on the order of gigabytes. How can it be cut down without seriously hurting quality?

The method

MQA's approach is extremely aggressive: all query heads share the same K and V.

In MHA:

Q: [batch, n_heads, seq_len, d_head]
K: [batch, n_heads, seq_len, d_head] ← n_heads copies
V: [batch, n_heads, seq_len, d_head] ← n_heads copies

In MQA:

Q: [batch, n_heads, seq_len, d_head] ← unchanged
K: [batch, 1, seq_len, d_head] ← just 1 copy!
V: [batch, 1, seq_len, d_head] ← just 1 copy!

The KV cache drops from n_heads copies to a single one, and memory bandwidth requirements fall drastically. Decoding speed improves substantially.

The cost: all heads now see the same "memory," losing the ability of different heads to attend to different subspaces. Quality degrades.

Key numbers

"Much faster to decode"
"Only minor quality degradation from the baseline"
Author: Noam Shazeer (one of the Transformer authors, later co-founder of Character.AI)

Impact assessment

MQA was the first step in "slimming down" attention. It proved an important principle: the KV cache is the inference bottleneck, not computation. All subsequent attention optimizations (GQA, MLA, SWA) revolve around "reducing the KV cache." But MQA's quality degradation meant it was not directly adopted by mainstream models — it was more of a thought experiment, proving that slimming is possible, and also that slimming too much hurts the brain.

Feynman-style takeaway

> MQA's real value is teaching you to identify the true bottleneck. Most people think Transformer inference is slow because attention involves heavy computation — no, what's slow is memory bandwidth. The attention matrix computation is O(n²), but each token only computes one row, which is actually fast. What's slow is moving the KV cache from GPU memory to the compute units. MQA is not "better attention"; it's "deleting the parts of attention that don't need to be copied." Like moving house: instead of throwing away furniture, you merge 96 identical chairs into 1.

---

arXiv: 1911.02150