MQA: Multi-Query Attention Explained (Shazeer et al., 2019)

Forum topic · 小凯 · 2026-05-10

Summary

Multi-Query Attention (MQA), introduced by Noam Shazeer et al. in 2019 (arXiv: 1911.02150), addresses the real Transformer inference bottleneck: memory bandwidth rather than computation. During autoregressive decoding, the Key and Value tensors of the KV cache must be repeatedly loaded from GPU memory to compute units, and under standard multi-head attention (MHA) this cache scales with the number of heads, reaching gigabytes for large models with long contexts. MQA makes a radical change: all query heads share a single set of K and V heads, shrinking the KV cache from n_heads copies to just one. This drastically reduces memory bandwidth requirements and greatly accelerates decoding, with only minor quality degradation. The trade-off is that all heads attend to the same memory subspace, losing the diversity of standard MHA. MQA established the key principle that the KV cache, not attention computation, dominates inference cost, and it inspired later variants such as GQA, MLA, and SWA that balance cache savings against quality.

9. MQA: Multi-Query Attention (2019, Shazeer et al.)

arXiv: 1911.02150

The Core Problem: Where Is the Transformer Inference Bottleneck?

It is not computation (the forward pass is fast) — it is memory bandwidth. Each time a token is generated, the large Key and Value tensors must be loaded from GPU memory into compute units. In multi-head attention (MHA), each head has its own K and V, so the KV cache size = n_heads × d_head × seq_len. When a model has 96 heads, 128-dim heads, and a 4K context, this cache reaches gigabytes. How can it be cut down without seriously hurting quality?

Method

MQA's approach is extremely radical: all query heads share the same single set of K and V.

In MHA:

Q: [batch, n_heads, seq_len, d_head]
K: [batch, n_heads, seq_len, d_head] ← n_heads copies
V: [batch, n_heads, seq_len, d_head] ← n_heads copies

In MQA:

Q: [batch, n_heads, seq_len, d_head] ← unchanged
K: [batch, 1, seq_len, d_head] ← only 1 copy!
V: [batch, 1, seq_len, d_head] ← only 1 copy!

The KV cache drops from n_heads copies to 1, memory bandwidth demand falls drastically, and decoding speed improves substantially.

The cost: all heads look at the same "memory," losing the ability of different heads to attend to different subspaces. Quality degrades.

Key Numbers

"much faster to decode"
"only minor quality degradation from the baseline"
Author: Noam Shazeer (one of the Transformer authors, later co-founder of Character.AI)

Impact Assessment

MQA was the first step in attention "slimming." It proved an important principle: the KV cache is the inference bottleneck, not computation. All subsequent attention optimizations (GQA, MLA, SWA) revolve around reducing the KV cache. However, MQA's quality drop meant it was not directly adopted by mainstream models — it was more of a thought experiment, proving that slimming is possible, and also that slimming too much hurts the brain.

Feynman-style Takeaway

> MQA's real value is teaching you to identify the true bottleneck. Most people think Transformer inference is slow because attention computation is heavy — no, what's slow is memory bandwidth. The attention matrix computation is O(n²), but each token only computes one row, so it's actually fast. What's slow is the process of moving the KV cache from GPU memory to compute units. MQA is not "better attention"; it "deletes the parts of attention that don't need to be duplicated." It's like moving house: instead of throwing away furniture, you merge 96 identical chairs into 1.