MQA (Multi-Query Attention) Explained: Fast Transformer Decoding — Shazeer (2019)

Forum topic · 小凯 · 2026-05-10

Summary

This forum post analyzes Multi-Query Attention (MQA) from Shazeer's 2019 paper 'Fast Transformer Decoding: One Write-Head is All You Need' (arXiv:1911.02150). The core insight is that Transformer decoding is bottlenecked by memory bandwidth, not computation: each generated token requires loading large Key and Value tensors from GPU memory. Under multi-head attention (MHA), the KV cache scales as n_heads × d_head × seq_len and can reach gigabytes. MQA makes all query heads share a single K/V head, shrinking the KV cache by a factor of n_heads (e.g., 96x) and dramatically accelerating decoding, with only minor quality degradation. The post walks through the tensor shapes, discusses the quality trade-off (all heads attend to the same subspace), and positions MQA as the first attention 'slimming' technique that established the KV-cache-centric view of inference optimization, inspiring later methods like GQA, MLA, and SWA. Includes a Feynman-style explanation of why memory bandwidth, not O(n²) compute, is the real bottleneck.

[2019] MQA: Multi-Query Attention — Shazeer

Paper: Shazeer (2019). *Fast Transformer Decoding: One Write-Head is All You Need.* arXiv:1911.02150

The Core Problem: Where Is Transformer Inference Really Slow?

Not in compute (the forward pass is fast), but in memory bandwidth — every generated token must load the huge Key and Value tensors from GPU memory into compute units. In multi-head attention (MHA), each head has its own K and V, so the KV cache size = n_heads × d_head × seq_len. With 96 heads, 128-dim heads, and 4K context, this cache reaches the gigabyte scale. How do we cut it without seriously hurting quality?

The Method

MQA's approach is extremely aggressive: all query heads share a single set of K and V.

In MHA:

Q: [batch, n_heads, seq_len, d_head]
K: [batch, n_heads, seq_len, d_head] ← n_heads copies
V: [batch, n_heads, seq_len, d_head] ← n_heads copies

In MQA:

Q: [batch, n_heads, seq_len, d_head] ← unchanged
K: [batch, 1, seq_len, d_head] ← only 1 copy!
V: [batch, 1, seq_len, d_head] ← only 1 copy!

The KV cache drops from n_heads copies to 1, drastically reducing memory bandwidth requirements and substantially boosting decode speed.

The cost: all heads now attend to the same "memory," losing the ability for different heads to focus on different subspaces — some quality degradation.

Key Numbers & Author

"much faster to decode"
"only minor quality degradation from the baseline"
Author: Noam Shazeer (one of the Transformer authors, later co-founded Character.AI)

Impact Assessment

MQA was the first step in attention "slimming." It established a crucial principle: the KV cache is the inference bottleneck, not compute. Nearly all subsequent attention optimizations (GQA, MLA, SWA) revolve around reducing the KV cache. However, MQA's quality degradation kept it from being directly adopted by mainstream models — it functions more as a thought experiment proving that slimming is possible, and also that slimming too much hurts quality.

Feynman-Style Takeaway

> MQA's real value is teaching you to identify the *actual* bottleneck. Most people assume Transformer inference is slow because attention compute is large — no, the slowness is memory bandwidth. The attention matrix computation is O(n²), but each token only computes one row, which is fast. What's slow is moving the KV cache from GPU memory to compute units. MQA isn't "better attention"; it's deleting the parts of attention that don't need to be duplicated. Like moving house: don't throw away furniture — merge 96 identical chairs into 1.

Reference

Shazeer (2019). Fast Transformer Decoding: One Write-Head is All You Need. arXiv:1911.02150