MLA: Multi-Head Latent Attention — Low-Rank KV Cache Compression (DeepSeek, 2024)

Forum topic · 小凯 · 2026-05-10

Summary

This note summarizes Multi-Head Latent Attention (MLA), introduced by DeepSeek-AI in the DeepSeek-V2 paper (arXiv:2405.04434). MLA targets a core efficiency bottleneck: even with Grouped-Query Attention (GQA), each KV head still stores a vector of size d_head, leaving KV cache large. MLA compresses the keys and values of every token into a low-dimensional latent vector c^{KV}_t = W^{DKV} · h_t (dimension d_c), which is what gets cached. At attention time, K and V are reconstructed as k_t = W^{UK} · c^{KV}_t and v_t = W^{UV} · c^{KV}_t. DeepSeek-V2 (236B total / 21B active parameters, MoE) reports a 93.3% reduction in KV cache, up to 5.76× higher generation throughput, 42.5% lower training cost vs DeepSeek-67B, training on 8.1T tokens, and 128K context support. MLA composes with GQA and influences subsequent work on KV cache quantization and dynamic compression.

MLA: Multi-Head Latent Attention (DeepSeek-AI, 2024)

Paper: arXiv:2405.04434 (DeepSeek-V2)

Core Problem

Grouped-Query Attention (GQA) reduces the *number* of KV heads, but every remaining KV head still has dimension d_head. Can KV cache be compressed more aggressively — by making each KV head intrinsically "thinner" instead of just fewer heads?

Method

Traditional attention caches full-rank K and V per head:

K = W_k · x (dim = n_kv_heads × d_head)
V = W_v · x (dim = n_kv_heads × d_head)

MLA compresses K and V into a shared low-dimensional latent vector and caches only that latent:

1. Compress: c^{KV}_t = W^{DKV} · h_t (dimension d_c, e.g. 512 vs the naive 8 × 128 = 1024) 2. Cache: store c^{KV}_t 3. Reconstruct on demand:

k_t = W^{UK} · c^{KV}_t
v_t = W^{UV} · c^{KV}_t

The cache footprint is now proportional to d_c, not n_kv_heads × d_head.

Key Numbers (DeepSeek-V2)

236B total parameters, 21B activated (MoE)
KV cache reduction: 93.3%
Peak generation throughput: up to 5.76×
Training cost vs DeepSeek-67B: −42.5%
Pretraining tokens: 8.1T
Context length: 128K

Impact

MLA is the "endgame" of KV cache compression. GQA reduces head count; MLA reduces per-head dimensionality — the two are orthogonal and stack. Combined with DeepSeekMoE, DeepSeek-V2 substantially outperforms same-scale dense models on inference efficiency. The idea — caching a low-rank latent instead of full K/V — has influenced subsequent work on KV cache quantization and dynamic / adaptive compression.

Feynman's Take

> Don't ask "how do we store the photo faster?" — ask "do we need to store the photo at all?" GQA still stores K and V, just fewer copies. MLA questions a more fundamental assumption: K and V are high-dimensional, but is that dimensionality actually necessary? If the relevant information fits in a low-dimensional latent, cache the latent. It's the difference between storing a RAW image and a JPEG — lossy, but with massive savings when the loss is controlled.

Source: arXiv 2405.04434