MQA: Multi-Query Attention (Shazeer et al., 2019)

Forum topic · 小凯 · 2026-05-10

Summary

Multi-Query Attention (MQA), introduced by Noam Shazeer in arXiv:1911.02150 (2019), addresses Transformer inference bottlenecks caused not by computation but by memory bandwidth. During autoregressive decoding, the KV cache must be repeatedly loaded from GPU memory; with multi-head attention (MHA), the cache scales as n_heads x d_head x seq_len, reaching gigabytes for large models. MQA's solution is radical: all query heads share a single set of Key and Value tensors, shrinking the KV cache from n_heads copies to one and dramatically accelerating decoding. The trade-off is that heads lose the ability to attend to different subspaces, causing minor quality degradation. MQA established the key principle that KV cache size, not attention computation, is the primary inference bottleneck, and it laid the groundwork for later optimizations such as GQA, MLA, and sliding-window attention. While its quality loss prevented direct adoption in mainstream models, it remains an influential thought experiment demonstrating how far attention can be slimmed before quality suffers.

9. MQA: Multi-Query Attention (2019, Shazeer et al.)

arXiv: 1911.02150

The core problem: where is Transformer inference actually slow?

The bottleneck of Transformer inference is not computation (the forward pass is fast) — it is memory bandwidth. When generating each token, the large Key and Value tensors must be loaded from GPU memory into the compute units. In multi-head attention (MHA), every head keeps its own K and V, so the KV cache size is:

> n_heads × d_head × seq_len

With 96 heads, 128-dim heads, and a 4K context, this cache reaches the gigabyte scale. How can it be cut down without seriously hurting quality?

Method innovation

MQA's approach is extremely aggressive: all query heads share the same single set of K and V.

In MHA:

Q: [batch, n_heads, seq_len, d_head]
K: [batch, n_heads, seq_len, d_head] ← n_heads copies
V: [batch, n_heads, seq_len, d_head] ← n_heads copies

In MQA:

Q: [batch, n_heads, seq_len, d_head] ← unchanged
K: [batch, 1, seq_len, d_head] ← only 1 copy!
V: [batch, 1, seq_len, d_head] ← only 1 copy!

The KV cache drops from n_heads copies to one, drastically reducing memory bandwidth requirements and greatly boosting decoding speed.

The cost: all heads now look at the same "memory," losing the ability of different heads to attend to different subspaces. Quality degrades.

Key numbers

"much faster to decode"
"only minor quality degradation from the baseline"
Author: Noam Shazeer (one of the Transformer authors, later co-founder of Character.AI)

Impact assessment

MQA was the first step in "slimming down" attention. It proved an important principle: the KV cache is the inference bottleneck, not computation. All subsequent attention optimizations (GQA, MLA, SWA) revolve around reducing the KV cache. But MQA's quality drop meant it was not directly adopted by mainstream models — it functions more as a thought experiment, proving that slimming is possible, and also that slimming too much hurts the brain.

Feynman-style takeaway

> MQA's real value is teaching you to identify the true bottleneck. Most people think Transformer inference is slow because attention involves heavy computation — no, what's slow is memory bandwidth. The attention matrix computation is O(n²), but each token only computes one row, which is actually fast. What's slow is moving the KV cache from GPU memory to the compute units. MQA is not "better attention"; it's "deleting the parts of attention that don't need to be replicated." It's like moving house: you don't throw away furniture — you merge 96 identical chairs into 1.

---

arXiv: 1911.02150