MQA: Multi-Query Attention — Shazeer (2019) Explained

Forum topic · 小凯 · 2026-05-10

Summary

This post analyzes Multi-Query Attention (MQA), proposed by Noam Shazeer in 2019 (arXiv:1911.02150), which addresses the Transformer inference bottleneck: memory bandwidth, not computation. During autoregressive decoding, the Key and Value tensors of the KV cache must be repeatedly loaded from GPU memory; with multi-head attention this cache scales as n_heads x d_head x seq_len, reaching gigabytes for large models. MQA keeps multiple query heads but makes all of them share a single K/V head, shrinking the KV cache by a factor equal to the number of heads and dramatically speeding up decoding with only minor quality degradation. The author argues MQA's core contribution is identifying the real bottleneck — bandwidth rather than attention's O(n²) computation — and establishing that KV-cache reduction is the key lever for fast inference. Although MQA itself was not widely adopted due to quality loss, it inspired follow-ups like GQA, MLA, and sliding-window attention. Includes a Feynman-style analogy: merging 96 identical chairs into one.

Background

This forum post reviews MQA (Multi-Query Attention) from Shazeer (2019), *Fast Transformer Decoding: One Write-Head is All You Need* (arXiv:1911.02150).

The Core Problem: Inference Is Bandwidth-Bound

The post argues that Transformer decoding is not bottlenecked by computation (the forward pass per token is fast), but by memory bandwidth: every generated token requires loading the large Key and Value tensors from GPU memory into the compute units.

With standard multi-head attention (MHA), each head has its own K and V, so the KV cache size = n_heads × d_head × seq_len. For a model with 96 heads, 128-dim heads, and 4K context, this cache reaches the gigabyte scale. How can it be cut without seriously hurting quality?

The Method

MQA's approach is radical: all query heads share a single set of K and V.

In MHA:

Q: [batch, n_heads, seq_len, d_head]
K: [batch, n_heads, seq_len, d_head] ← n_heads copies
V: [batch, n_heads, seq_len, d_head] ← n_heads copies

In MQA:

Q: [batch, n_heads, seq_len, d_head] ← unchanged
K: [batch, 1, seq_len, d_head] ← only 1 copy!
V: [batch, 1, seq_len, d_head] ← only 1 copy!

The KV cache drops from n_heads copies to one, drastically reducing memory bandwidth needs and greatly speeding up decoding.

Cost: all heads see the same "memory", losing the ability of different heads to attend to different subspaces — a quality drop.

Key Numbers / Claims

"Much faster to decode"
"Only minor quality degradation from the baseline"
Author: Noam Shazeer (one of the Transformer authors, later at Character.AI)

Impact

MQA was the first step in attention "slimming". It established an important principle: the KV cache is the inference bottleneck, not computation. Subsequent attention optimizations (GQA, MLA, SWA) all revolve around reducing the KV cache. But MQA's quality degradation kept it from direct mainstream adoption — it is more of a "thought experiment" proving slimming is possible, and that slimming too much hurts quality.

Feynman-Style Takeaway (translated)

> MQA's real value is teaching you to identify the true bottleneck. Most people think Transformer inference is slow because attention's computation is heavy — no, what's slow is memory bandwidth. The attention matrix computation is O(n²), but each token only computes one row, which is actually fast. The slow part is moving the KV cache from GPU memory to the compute units. MQA is not "better attention"; it removes the parts of attention that don't need to be duplicated. Like moving house: instead of throwing away furniture, you merge 96 identical chairs into one.

Reference

Shazeer (2019). *Fast Transformer Decoding: One Write-Head is All You Need.* arXiv:1911.02150