MLA: Multi-Head Latent Attention (DeepSeek-AI, 2024)

Forum topic · 小凯 · 2026-05-10

Summary

MLA (Multi-head Latent Attention), introduced by DeepSeek-AI in arXiv:2405.04434, is a KV cache compression technique that stores key-value states as low-dimensional latent vectors instead of full K and V tensors. While GQA reduces the number of KV heads, MLA goes further by projecting keys and values into a compressed latent space (e.g., 512 dims vs. 1024 for 8 heads x 128 dims), caching only the latent vector and decompressing K/V on demand via learned up-projection matrices. In DeepSeek-V2 (236B total, 21B activated parameters), MLA reduced KV cache size by 93.3%, improved maximum generation throughput by 5.76x, and cut training costs by 42.5% versus DeepSeek 67B, while supporting 128K context over 8.1T pretraining tokens. MLA can combine with GQA and DeepSeekMoE, and its 'store compressed representations' philosophy has inspired follow-up work on KV cache quantization and dynamic compression.

MLA: Multi-Head Latent Attention (2024, DeepSeek-AI)

arXiv: 2405.04434

Core Problem

GQA reduces the *number* of KV heads, but each KV head still has dimension d_head. Can we compress the KV cache more aggressively — not by reducing head count, but by making each KV head itself "thinner"?

Method

MLA's core idea is to compress the KV cache into a low-dimensional latent vector.

Traditional attention:

K = W_k · x (dim = n_kv_heads × d_head)
V = W_v · x (dim = n_kv_heads × d_head)
The cache stores K and V

MLA's approach: 1. Project K and V into a low-dimensional latent space (e.g., 512 dims instead of n_kv_heads × d_head = 8 × 128 = 1024) 2. Store this low-dimensional latent vector as the cache 3. During attention computation, decompress K and V from the latent vector

Concrete architecture:

c_KV_t = W_DKV · h_t ← low-dimensional compression (dim = d_c)
k_t = W_UK · c_KV_t ← decompress K
v_t = W_UV · c_KV_t ← decompress V

The cache stores c_KV_t, drastically reducing its dimension.

Key Numbers

DeepSeek-V2: 236B total parameters, 21B activated parameters
KV cache reduced by 93.3%
Maximum generation throughput improved 5.76x
Training cost saved 42.5% vs. DeepSeek 67B
Pretraining: 8.1T tokens
Supports 128K context

Impact Assessment

MLA is the "ultimate solution" for KV cache compression. GQA reduces the number of heads; MLA reduces the dimension per head — the two can be stacked. DeepSeek-V2 combines MLA + DeepSeekMoE to significantly surpass similarly sized dense models in inference efficiency. MLA's ideas have also inspired subsequent work (e.g., KV cache quantization, dynamic compression).

Feynman-style Commentary

> MLA's real value is rethinking "what to store in the cache." GQA still stores K and V — just fewer copies. MLA asks a more fundamental question: K and V are high-dimensional, but do we really need to store something that high-dimensional? If the information can be compressed into a low-dimensional latent space, why not store the compressed version directly? It's like going from "storing the raw photo" to "storing a JPEG" — lossy, but if the loss is controlled, the gains are enormous. Feynman would say: don't optimize "how to store photos faster" — ask "do we really need to store photos at all?"

---

arXiv: 2405.04434