MLA: Multi-Head Latent Attention (DeepSeek-AI, 2024) — Compressing the KV Cache into a Low-Rank Latent Vector

Forum topic · 小凯 · 2026-05-10

Summary

MLA (Multi-head Latent Attention), introduced by DeepSeek-AI in arXiv:2405.04434, is a KV cache compression technique that goes beyond grouped-query attention (GQA). Instead of reducing the number of KV heads, MLA compresses keys and values into a low-dimensional latent vector: at each token, a down-projection matrix W^{DKV} maps the hidden state to a latent c^{KV}_t (e.g., 512 dims versus 1024 dims for 8 heads of 128), and only this latent is cached. Keys and values are reconstructed on the fly via up-projection matrices W^{UK} and W^{UV} during attention computation. In DeepSeek-V2 (236B total / 21B activated parameters), MLA reduced the KV cache by 93.3%, boosted maximum generation throughput by 5.76x, cut training costs by 42.5% versus DeepSeek 67B, was pretrained on 8.1T tokens, and supports 128K context. The forum post frames MLA as a fundamental rethink of what the cache should store—save the compressed representation rather than full K/V, analogous to storing JPEGs instead of raw photos—and notes MLA can stack with GQA-style and MoE techniques, inspiring later KV quantization and dynamic compression work.

MLA: Multi-Head Latent Attention (2024, DeepSeek-AI)

Paper: arXiv:2405.04434

The Core Problem

GQA reduces the *number* of KV heads, but each KV head still has dimension d_head. Is there a way to compress the KV cache more aggressively—not by reducing the head count, but by making each KV head itself "thinner"?

The Method

MLA's core idea is to compress the KV cache into a low-dimensional latent vector.

Traditional attention:

K = W_k · x (dim = n_kv_heads × d_head)
V = W_v · x (dim = n_kv_heads × d_head)
The cache stores K and V directly.

MLA's approach: 1. Compress K and V into a low-dimensional latent space (e.g., 512 dims, instead of n_kv_heads × d_head = 8 × 128 = 1024 dims) 2. Store this low-dimensional latent vector as the cache 3. During attention computation, decompress the required K and V from the latent vector

The architecture:

c^{KV}_t = W^{DKV} · h_t ← low-dimensional compression (dim = d_c)
k_t = W^{UK} · c^{KV}_t ← decompress K
v_t = W^{UV} · c^{KV}_t ← decompress V

The cache stores c^{KV}_t, whose dimension is dramatically smaller.

Key Numbers (DeepSeek-V2)

236B total parameters, 21B activated parameters
KV cache reduced by 93.3%
Maximum generation throughput improved by 5.76x
Training cost saved 42.5% compared to DeepSeek 67B
Pretraining: 8.1T tokens
Supports 128K context

Impact Assessment

MLA is the "ultimate solution" for KV cache compression. GQA reduces the number of heads; MLA reduces the dimensionality of what each head needs to store—the two can be combined. DeepSeek-V2 pairs MLA with DeepSeekMoE to substantially outperform similarly sized dense models in inference efficiency. MLA has also inspired follow-up work (e.g., KV cache quantization, dynamic compression).

Commentary

MLA's real value is rethinking *what the cache stores*. GQA still stores K and V—just fewer copies. MLA asks a more fundamental question: K and V are high-dimensional, but do we really need to store something that high-dimensional? If the information can be compressed into a low-dimensional latent space, why not store the compressed version directly? It's like moving from "storing raw photos" to "storing JPEGs"—lossy, but if the loss is controlled, the gains are enormous. As Feynman would say: don't optimize "how to store photos faster"—ask "do we actually need to store the photos at all?"

Paper: arXiv:2405.04434