MQA: Multi-Query Attention Explained (Shazeer et al., 2019)

Forum topic · 小凯 · 2026-05-10

Summary

Multi-Query Attention (MQA), introduced by Noam Shazeer et al. in 2019 (arXiv: 1911.02150), addresses the real inference bottleneck of Transformers: not computation, but memory bandwidth. During autoregressive decoding, large Key and Value tensors from the KV cache must be repeatedly loaded from GPU memory; with many heads (e.g., 96 heads at 128 dimensions and 4K context), this cache reaches gigabytes. MQA's radical solution is to let all query heads share a single set of K and V projections, shrinking the KV cache from n_heads copies to just one, which drastically cuts bandwidth and greatly accelerates decoding. The trade-off is a minor quality degradation, since heads lose the ability to attend to different subspaces. MQA established the key principle that the KV cache, not attention math, dominates inference cost, paving the way for GQA, MLA, and other cache-reduction techniques. Though too aggressive for direct mainstream adoption, it remains a foundational thought experiment in attention efficiency.

MQA: Multi-Query Attention (2019, Shazeer et al.)

Paper: arXiv: 1911.02150

The Core Problem

Where is the inference bottleneck in Transformers? Not in computation (the forward pass is fast), but in memory bandwidth — every generated token requires loading huge Key and Value tensors from GPU memory to the compute units.

In Multi-Head Attention (MHA), each head has its own K and V, so the KV cache size = n_heads × d_head × seq_len. With 96 heads, 128-dim heads, and 4K context, this cache is on the order of gigabytes. How do you shrink it without seriously hurting quality?

The Method

MQA's approach is extremely aggressive: all query heads share the same single set of K and V.

In MHA:

Q: [batch, n_heads, seq_len, d_head]
K: [batch, n_heads, seq_len, d_head] ← n_heads copies
V: [batch, n_heads, seq_len, d_head] ← n_heads copies

In MQA:

Q: [batch, n_heads, seq_len, d_head] ← unchanged
K: [batch, 1, seq_len, d_head] ← just 1 copy!
V: [batch, 1, seq_len, d_head] ← just 1 copy!

The KV cache drops from n_heads copies to 1, drastically reducing memory bandwidth requirements and substantially boosting decoding speed.

The cost: all heads look at the same "memory," losing the ability of different heads to focus on different subspaces. Quality degrades.

Key Numbers

"much faster to decode"
"only minor quality degradation from the baseline"
Author: Noam Shazeer (one of the Transformer authors, later co-founder of Character.AI)

Impact Assessment

MQA was the first step in attention "slimming." It proved an important principle: the KV cache is the inference bottleneck, not computation. All subsequent attention optimizations (GQA, MLA, SWA) revolve around reducing the KV cache. But MQA's quality degradation meant it was not directly adopted by mainstream models — it's more of a "thought experiment," proving that slimming is possible, and also that slimming too much hurts.

Feynman-Style Takeaway

> MQA's real value is teaching you to identify the true bottleneck. Most people think Transformer inference is slow because attention computation is heavy — no, what's slow is memory bandwidth. The attention matrix computation is O(n²), but each token only computes one row, so it's actually fast. What's slow is the process of moving the KV cache from GPU memory to the compute units. MQA isn't "better attention" — it's "deleting the parts of attention that don't need to be duplicated." It's like moving house: you don't throw away the furniture, you merge 96 identical chairs into 1.

---

Paper: arXiv: 1911.02150