[2024] MLA: Multi-Head Latent Attention — DeepSeek-AI Explained

Forum topic · 小凯 · 2026-05-10

Summary

This post explains Multi-Head Latent Attention (MLA), the KV cache compression technique introduced by DeepSeek-AI in the DeepSeek-V2 paper (arXiv:2405.04434). While Grouped-Query Attention (GQA) reduces the number of KV heads, MLA goes further by compressing keys and values into a low-dimensional latent vector (e.g., 512 dims vs. 1024 dims for 8 heads × 128 dims). Only the latent vector is cached; K and V are reconstructed on the fly via up-projection matrices. Reported results for DeepSeek-V2 (236B total, 21B activated parameters): 93.3% KV cache reduction, 5.76x maximum generation throughput improvement, 42.5% training cost savings versus DeepSeek 67B, 8.1T token pretraining, and 128K context support. The author argues MLA is the definitive KV cache compression approach—reducing per-head dimension rather than head count—and can combine with GQA and DeepSeekMoE. A Feynman-style commentary frames MLA as rethinking 'what to cache,' analogous to storing JPEGs instead of raw photos.

Core Problem

GQA reduces the number of KV heads, but each KV head still has dimension d_head. Can we compress the KV cache more aggressively—not by reducing head count, but by making each KV head itself "thinner"?

Method: Compress KV into a Low-Dimensional Latent Vector

Traditional attention:

K = W_k · x (dim = n_kv_heads × d_head)
V = W_v · x (dim = n_kv_heads × d_head)
The cache stores K and V directly

MLA's approach:

1. Compress K and V into a low-dimensional latent space (e.g., 512 dims instead of n_kv_heads × d_head = 8 × 128 = 1024 dims) 2. Store this low-dimensional latent vector as the cache 3. During attention computation, decompress the needed K and V from the latent vector

Architecture:

c^{KV}_t = W^{DKV} · h_t — low-dimensional compression (dim = d_c)
k_t = W^{UK} · c^{KV}_t — up-project to get K
v_t = W^{UV} · c^{KV}_t — up-project to get V

The cache stores c^{KV}_t, dramatically reducing its dimensionality.

Key Numbers (DeepSeek-V2)

236B total parameters, 21B activated parameters
93.3% KV cache reduction
5.76x maximum generation throughput improvement
42.5% training cost savings vs. DeepSeek 67B
Pretrained on 8.1T tokens
Supports 128K context

Impact Assessment

MLA is arguably the "ultimate" KV cache compression method. GQA reduces the number of heads; MLA reduces the dimensionality of what's stored per head — the two are composable. DeepSeek-V2 combines MLA + DeepSeekMoE to substantially outperform similarly sized dense models in inference efficiency. MLA has also inspired follow-up work (KV cache quantization, dynamic compression, etc.).

Feynman-Style Commentary

> MLA's real value is rethinking "what the cache stores." GQA still stores K and V — just fewer copies. MLA asks a more fundamental question: K and V are high-dimensional, but do we really need to store something that high-dimensional? If the information can be compressed into a low-dimensional latent space, why not store the compressed version directly? It's like moving from "storing raw photos" to "storing JPEGs" — lossy, but if the loss is controllable, the gains are huge. Feynman would say: don't optimize "how to store photos faster"; ask "do we really need to store photos at all?"

Reference

DeepSeek-AI (2024). DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model. arXiv:2405.04434