MLA: Multi-Head Latent Attention — How DeepSeek Compressed KV Cache by 93.3%

Forum topic · 小凯 · 2026-05-10

Summary

This forum post explains Multi-Head Latent Attention (MLA), the KV cache compression technique introduced by DeepSeek-AI in the DeepSeek-V2 paper (arXiv:2405.04434). While Grouped-Query Attention (GQA) reduces the number of KV heads, each KV head remains full-dimensional. MLA goes further by compressing keys and values into a low-dimensional latent vector (e.g., 512 dimensions versus 1024), caching only this latent representation, and decompressing K and V on demand during attention computation. In DeepSeek-V2 (236B total parameters, 21B activated per token), MLA reduced KV cache size by 93.3%, improved maximum generation throughput by 5.76x, and cut training cost by 42.5% versus DeepSeek 67B, while supporting 128K context on 8.1T pretraining tokens. The post includes the architecture formulas, key performance numbers, and an analysis arguing MLA's core insight is rethinking what should be cached—storing compressed latent representations rather than full K/V matrices. MLA combines with DeepSeekMoE to outperform similarly sized dense models in inference efficiency.

MLA: Multi-Head Latent Attention — DeepSeek-AI

Paper: arXiv:2405.04434

The Core Problem

GQA reduces the *number* of KV heads, but each KV head still has dimension d_head. Can we compress the KV cache more aggressively — not by reducing head count, but by making each KV head itself "thinner"?

The Method

MLA's core idea is to compress the KV cache into a low-dimensional latent vector.

Traditional attention:

K = W_k · x (dim = n_kv_heads × d_head)
V = W_v · x (dim = n_kv_heads × d_head)
The cache stores K and V directly

MLA's approach: 1. Compress K and V into a low-dimensional latent space (e.g., 512 dims, instead of n_kv_heads × d_head = 8 × 128 = 1024 dims) 2. Cache this low-dimensional latent vector 3. During attention computation, decompress the needed K and V from the latent vector

Architecture:

c^{KV}_t = W^{DKV} · h_t ← low-dimensional compression (dim = d_c)
k_t = W^{UK} · c^{KV}_t ← decompress K
v_t = W^{UV} · c^{KV}_t ← decompress V

The cache stores only c^{KV}_t, drastically reducing its dimensionality.

Key Numbers (DeepSeek-V2)

236B total parameters, 21B activated per token
KV cache reduced by 93.3%
Maximum generation throughput improved by 5.76x
Training cost 42.5% lower than DeepSeek 67B
Pretrained on 8.1T tokens
Supports 128K context

Impact Assessment

MLA is arguably the "ultimate solution" for KV cache compression. GQA reduces head count; MLA reduces the per-head dimensionality — and the two can be stacked. DeepSeek-V2 combines MLA with DeepSeekMoE to substantially outperform similarly sized dense models in inference efficiency. MLA's ideas have also inspired follow-up work (e.g., KV cache quantization, dynamic compression).

Commentary (Feynman-style)

> MLA's real value is rethinking *what to cache*. GQA still stores K and V — just fewer copies. MLA asks a more fundamental question: K and V are high-dimensional, but do we really need to store something that high-dimensional? If the information can be compressed into a low-dimensional latent space, why not store the compressed version directly? It's like moving from "storing the raw photo" to "storing the JPEG" — lossy, but if the loss is controllable, the gains are enormous. Feynman would say: don't optimize "how to store photos faster"; ask "do we even need to store the photo?"

Reference

DeepSeek-AI (2024). DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model. arXiv:2405.04434