MQA: Multi-Query Attention (Shazeer, 2019) — Shrinking the KV Cache by Sharing Keys and Values

Forum topic · 小凯 · 2026-05-10

Summary

Multi-Query Attention (MQA), introduced by Noam Shazeer in 2019 (arXiv 1911.02150), addresses the real bottleneck of Transformer inference: memory bandwidth rather than computation. In standard multi-head attention (MHA), each head maintains its own Key and Value tensors, so the KV cache scales with the number of heads and can reach gigabytes for large models with long contexts. MQA makes an aggressive change: all query heads share a single set of K and V tensors, reducing the KV cache from n_heads copies to one. This drastically lowers memory traffic and greatly accelerates autoregressive decoding, with only minor quality degradation according to the original paper. The trade-off is that all heads see the same memory, losing the ability to attend to diverse subspaces, which prevented direct adoption by mainstream models. MQA's lasting contribution is conceptual: it established that the KV cache, not FLOPs, is the inference bottleneck, laying the groundwork for successors like GQA, MLA, and sliding-window attention that balance cache reduction against quality.

MQA: Multi-Query Attention (2019, Shazeer et al.)

arXiv: 1911.02150

The Core Problem: Where Is the Transformer Inference Bottleneck?

Not in computation (the forward pass is fast), but in memory bandwidth — every generated token requires loading huge Key and Value tensors from GPU memory to compute units. In multi-head attention (MHA), each head has its own K and V, so the KV cache size = n_heads × d_head × seq_len. With 96 heads, 128-dim heads, and 4K context, this cache reaches gigabytes. How can it be cut without seriously hurting quality?

The Method

MQA's approach is radically simple: all query heads share one set of K and V.

In MHA:

Q: [batch, n_heads, seq_len, d_head]
K: [batch, n_heads, seq_len, d_head] ← n_heads copies
V: [batch, n_heads, seq_len, d_head] ← n_heads copies

In MQA:

Q: [batch, n_heads, seq_len, d_head] ← unchanged
K: [batch, 1, seq_len, d_head] ← only 1 copy!
V: [batch, 1, seq_len, d_head] ← only 1 copy!

The KV cache drops from n_heads copies to 1, drastically reducing memory bandwidth and greatly speeding up decoding.

Cost: all heads share the same "memory," losing the ability to attend to different subspaces. Quality degrades.

Key Numbers

"Much faster to decode"
"Only minor quality degradation from the baseline"
Author: Noam Shazeer (one of the Transformer authors, later co-founded Character.AI)

Impact Assessment

MQA was the first step in attention "slimming." It proved a key principle: the KV cache is the inference bottleneck, not computation. All subsequent attention optimizations (GQA, MLA, SWA) revolve around reducing the KV cache. But MQA's quality drop meant it was not directly adopted by mainstream models — it functions more as a thought experiment, proving slimming is possible, and that slimming too much hurts.

Feynman-Style Takeaway

> MQA's real value is teaching you to identify the true bottleneck. Most people think Transformer inference is slow because of attention computation — no, it's memory bandwidth. The attention matrix computation is O(n²), but each token only computes one row, which is actually fast. The slow part is moving the KV cache from VRAM to compute units. MQA is not "better attention"; it's "deleting the parts of attention that don't need to be duplicated." It's like moving house: instead of throwing out furniture, you merge 96 identical chairs into 1.

---

arXiv: 1911.02150