MLA: Multi-Head Latent Attention (DeepSeek-AI, 2024) Explained

Forum topic · 小凯 · 2026-05-10

Summary

MLA (Multi-head Latent Attention), introduced by DeepSeek-AI in arXiv 2405.04434 (2024), is a KV cache compression technique for transformer inference. Unlike GQA, which reduces the number of KV heads, MLA compresses keys and values into a low-dimensional latent vector (e.g., 512 dimensions versus 1024 for n_kv_heads × d_head), stores only this latent in the cache, and decompresses K and V on the fly via up-projection matrices during attention. In DeepSeek-V2 (236B total parameters, 21B activated), MLA achieved a 93.3% reduction in KV cache size and up to 5.76x higher maximum generation throughput, with 42.5% training cost savings versus DeepSeek 67B, 8.1T pretraining tokens, and 128K context support. The post frames MLA as a fundamental rethinking of what should be cached: rather than optimizing how to store full-resolution K and V, MLA asks whether high-dimensional K/V representations can be lossily compressed like JPEG images—accepting controlled information loss for major efficiency gains.

MLA: Multi-head Latent Attention (2024, DeepSeek-AI)

arXiv: 2405.04434

Core Problem

GQA reduces the *number* of KV heads, but each KV head still has dimension d_head. Can we compress the KV cache even further—not by reducing the head count, but by making each KV head itself "thinner"?

Method

MLA's core idea is to compress the KV cache into a low-dimensional latent vector.

Traditional attention:

K = W_k · x (dim = n_kv_heads × d_head)
V = W_v · x (dim = n_kv_heads × d_head)
The cache stores K and V

MLA's approach: 1. Compress K and V into a low-dimensional latent space (e.g., 512 dims instead of n_kv_heads × d_head = 8 × 128 = 1024 dims) 2. Store this low-dimensional latent vector as the cache 3. During attention, decompress the required K and V from the latent vector

Concrete architecture:

c_t^{KV} = W^{DKV} · h_t ← low-dimensional compression (dim = d_c)
k_t = W^{UK} · c_t^{KV} ← decompress K
v_t = W^{UV} · c_t^{KV} ← decompress V

The cache stores c_t^{KV}, whose dimension is substantially smaller.

Key Numbers

DeepSeek-V2: 236B total parameters, 21B activated parameters
KV cache reduced by 93.3%
Maximum generation throughput improved by 5.76x
Training cost saved 42.5% versus DeepSeek 67B
Pretraining: 8.1T tokens
Supports 128K context

Impact Assessment

MLA is the "ultimate solution" for KV cache compression. GQA reduces the head count; MLA reduces the dimension per head—the two can be combined. DeepSeek-V2 pairs MLA with DeepSeekMoE, substantially outperforming dense models of similar scale in inference efficiency. MLA's idea has also inspired follow-up work (e.g., KV cache quantization, dynamic compression).

Feynman-Style Commentary

> MLA's real value is rethinking *what to store in the cache*. GQA still stores K and V—just fewer copies. MLA asks a more fundamental question: K and V are high-dimensional, but do we really need to store something that high-dimensional? If the information can be compressed into a low-dimensional latent space, why not store the compressed version directly? It's like going from "storing the raw photo" to "storing a JPEG"—lossy, but if the loss is controlled, the gains are huge. Feynman would say: don't optimize "how to store photos faster"; ask "do we really need to store photos at all?"

---

arXiv: 2405.04434