How DeepSeek Rewrote the Transformer: MLA's 57x KV Cache Compression Magic

Forum topic · 小凯 · 2026-06-08

Summary

This post explains Multi-head Latent Attention (MLA), the memory-efficiency innovation behind DeepSeek-V2, V3, and R1. Traditional multi-head attention stores full Key/Value (KV) caches for every token; for a model like DeepSeek-R1 this amounts to roughly 4MB per token, making long-context inference impractical (40GB at 10K tokens). Earlier fixes like Multi-Query Attention (MQA) and Grouped-Query Attention (GQA) reduce KV cache size by sharing heads, but at a quality cost. MLA instead applies low-rank compression: it caches a compact latent vector (e.g., 512 dims instead of 4096) and reconstructs multi-head KV via learned up-projections during attention computation. Absorbed projections use the associativity of matrix multiplication to fold decompression matrices into query/output projections, skipping explicit decompression with exact mathematical equivalence. Decoupled RoPE preserves positional information through a separate small pathway. Per Welch Labs' calculations based on the DeepSeek-V2/V3 papers, MLA cuts KV cache parameters from 3,997,696 to 70,272 per token — a 56.9x reduction (93.3% per DeepSeek-V2), while boosting throughput by nearly 6x. The article covers the MHA→MQA→GQA→MLA evolution, hardware implications (memory-bound to compute-bound), limitations, and references including arXiv:2405.04434, arXiv:2412.19437, arXiv:2501.12948, and arXiv:2502.14837.

How DeepSeek Rewrote the Transformer: MLA's 57x Compression Magic

> Original video: *How DeepSeek Rewrote the Transformer [MLA]* by Welch Labs > Video: http://www.youtube.com/watch?v=kYI-U1_I3mQ > Papers: DeepSeek-V2 (arXiv:2405.04434), DeepSeek-V3 (arXiv:2412.19437), DeepSeek-R1 (arXiv:2501.12948)

---

Introduction: The 4MB-per-token Disaster

Imagine chatting with a large model. Every word it generates requires storing a piece of "memory" in GPU VRAM — the Key and Value vectors of the attention mechanism (the KV cache).

For a model at the scale of DeepSeek-R1, using traditional multi-head attention (MHA), each generated token costs about 4MB of KV cache.

4MB? Sounds small?

But with a 10K-token context, that's 40GB — already beyond most consumer GPUs. At 100K tokens? 400GB. You'd need an entire server for one long conversation.

This is the fatal bottleneck of the traditional Transformer: the KV cache grows linearly with both context length and model size — a double explosion.

DeepSeek's answer is MLA (Multi-head Latent Attention). Instead of buying bigger GPUs, it improved the Transformer's memory efficiency by 57x (estimated for V3/R1).

That's like compressing a whole data center's VRAM requirement onto a single GPU.

---

Why Does the KV Cache Explode? Starting from MHA

Traditional Multi-Head Attention (MHA)

In every Transformer layer, the model has multiple attention heads (e.g., 64), and each head maintains its own set of Key and Value vectors.

Assume:

$$L$$ layers
$$h$$ attention heads per layer
Per-head dimension $$d_k$$
Sequence length $$N$$

Then the total KV cache size is:

\text{KV Cache} = 2 \times L \times h \times d_k \times N \times \text{bytes per param}

The factor of 2 accounts for Keys and Values. For DeepSeek-R1 (671B parameters, 37B active), standard MHA would yield roughly 3,997,696 cached parameters per token, per Welch Labs' calculation.

From MHA to MQA/GQA: Cutting Heads for Memory

MQA (Multi-Query Attention): all Query heads share 1 KV head. KV cache drops to $$1/h$$ .
GQA (Grouped-Query Attention): Query heads are grouped, each group sharing a few KV heads. Cache drops to $h_{group}/h$ .

These reduce memory, but quality suffers — shared KV heads can't capture the diverse information different Query heads need. Like 64 journalists sharing one recorder: cheap, but information is lost.

DeepSeek said: don't cut heads — compress.

---

MLA's Core Insight: Low-Rank Compression

Don't store the "full version" of KV in the cache — store a highly compressed "latent vector", and decompress it when needed.

Analogy:

MHA stores every uncompressed HD frame on disk
MQA/GQA keeps fewer copies, but at full resolution
MLA compresses the movie to MP4 and decodes in real time at playback

The Math

Instead of storing $K_t, V_t \in \mathbb{R}^{d_k}$ , MLA stores a low-dimensional latent vector:

c_t^{KV} = [W^{DK} K_t; W^{DV} V_t] \in \mathbb{R}^{d_c}

where $d_c \ll d_k$ (e.g., $$d_c = 512$$ vs. $$d_k = 4096$$ ). $W^{DK}$ and $W^{DV}$ are down-projection matrices.

During attention, up-projection matrices reconstruct multi-head KV:

\tilde{K}_t = W^{UK} c_t^{KV}, \quad \tilde{V}_t = W^{UV} c_t^{KV}

How Dramatic Is It?

The DeepSeek-V2 paper claims a 93.3% KV cache reduction. Welch Labs recalculated from V3/R1's architecture and estimated a 56.9x reduction (from 3,997,696 to 70,272 parameters).

Either way: 128K context runs comfortably on ordinary GPUs.

---

Absorbed Projections: A Mathematician's "Lazy Trick"

The Redundant Decompression Step

The naive MLA pipeline:

1. Decompress: $\tilde{K} = W^{UK} c^{KV}$ 2. Compute attention scores: $Q \tilde{K}^T$ 3. Weighted sum with $\tilde{V}$ 4. Output projection: $O = W^O \cdot \text{AttentionOutput}$

The Trick: Associativity of Matrix Multiplication

Since $W^{UK}$ , $W^{UV}$ , and $$W^O$$ are fixed weight matrices, they can be "absorbed" into other matrices:

1. Query-side absorption: $W^{UQ}$ can merge with $W^{UK}$ along the shared decompression path. 2. Output-side absorption: $W^{UV}$ and $$W^O$$ merge into one matrix. 3. V reused from K: Value vectors need not be stored separately — they are taken from the first $$d_c$$ dimensions of the Key vector.

As Welch Labs puts it:

> "This means during inference the model can skip the explicit decompression step and perform part of the computation directly in the compressed low-dimensional space."

This is not approximation — it's exact mathematical equivalence. Rearranging multiplication order eliminates intermediate computations with identical results, at greatly reduced cost.

---

The Dance with RoPE: Decoupled Positional Encoding

MLA has a hidden challenge: RoPE (Rotary Position Embedding). RoPE injects positional information into Q and K, but MLA computes Q/K in a low-dimensional latent space before decompression — applying RoPE directly to low-dimensional vectors would distort positions after decompression.

DeepSeek's solution is Decoupled RoPE:

For RoPE-participating dimensions, Q and K keep a small, independent position-encoded vector
Non-RoPE dimensions go through the low-rank compression path
The two are concatenated for use

Positional information travels through a dedicated "express lane", compressed information through the low-rank path, without interference. (The Welch Labs video simplified this — noted in its author's remarks.)

---

Why Is DeepSeek's API So Absurdly Cheap?

1. Fewer GPUs serve a bigger model

2. Faster inference

just under 6x

more than 6x

3. Memory bandwidth is no longer the bottleneck

memory bandwidth

> The model didn't get smaller — it got smarter: it compresses memory to the extreme and decompresses on demand.

---

The Evolution of Attention: MHA → MLA

| Mechanism | Core idea | KV cache | Quality cost | Representative models | |:---|:---|:---|:---|:---| | MHA | Each Query head has its own KV head | $2 \times h \times d_k$ | Baseline | GPT-3, LLaMA-1 | | MQA | All Query heads share 1 KV head | $2 \times d_k$ | Significant drop | PaLM, Falcon | | GQA | Query heads grouped, share KV heads | $2 \times g \times d_k$ | Moderate drop | LLaMA-2, Mistral | | MLA | Low-rank compression + absorbed projections | $2 \times d_c$ ( $d_c \ll d_k$ ) | Almost none | DeepSeek-V2/3/R1 |

MLA's breakthrough: it trades memory via compression, not head-cutting — preserving multi-head expressiveness while approaching MQA-level storage.

---

Hardware Perspective: MLA Is Architecture, Not Just an Algorithm

A 2025 hardware analysis paper (Robin Geens, KU Leuven) notes:

> MLA not only reduces bandwidth usage but can shift attention computation from memory-bound to compute-bound.

Future AI accelerators optimized for MLA could be an order of magnitude faster than MHA-optimized ones. MLA is a hardware-software co-design opportunity.

---

Limitations and Open Questions

1. Higher training cost: compression matrices must be learned during pretraining (zero cost at inference). 2. Ecosystem compatibility: MLA is DeepSeek-specific; mainstream frameworks (HuggingFace, vLLM) need dedicated support — FlashMLA kernels were written by DeepSeek itself. 3. Flexibility of attention patterns: whether fixed compression/decompression matrices limit learned attention patterns remains unclear. 4. Ultra-long contexts (>128K): whether decompression errors accumulate at 1M+ tokens needs more research.

---

Conclusion: It's a Philosophy, Not Just Parameters

Traditional Transformers say: memory must be stored verbatim. MQA/GQA say: memory can be shared, at the cost of detail. MLA says: memory can be compressed, as long as it's reconstructed exactly at computation time.

The 57x compression isn't magic — it's math: low-rank approximation, matrix absorption, decoupled RoPE. Each is rigorous linear algebra; combined, they create a qualitative shift.

A deeper question: if an LLM's memory can be compressed 57x without precision loss, what's the human brain's compression ratio? We don't remember every detail of events — we store compressed "semantic vectors" and decompress details on demand.

Next step: could entire weight layers be compressed into latent space? Could all of a model's knowledge become a set of always-decompressible latent vectors? If so, the Transformer wouldn't just be rewritten — it would be redefined.

---

References

Video: How DeepSeek Rewrote the Transformer [MLA] by Welch Labs — http://www.youtube.com/watch?v=kYI-U1_I3mQ
Papers:
DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model (arXiv:2405.04434)
DeepSeek-V3 (arXiv:2412.19437)
DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning (arXiv:2501.12948)
Hardware analysis: Hardware-Centric Analysis of DeepSeek's Multi-Head Latent Attention (Robin Geens, KU Leuven, 2025)
Migration method: MHA2MLA: Towards Economical Inference (arXiv:2502.14837) — adapts MHA models like Llama to MLA with only 0.3%–0.6% of data, reducing KV cache by 92.19%
Video author's notes:

1. DeepSeek-V2 paper claims 93.3% KV cache reduction 2. V3/R1 estimated at 56.9x (3,997,696/70,272) 3. Throughput gain: just under 6x per V2 paper; possibly larger for V3/R1 4. Beginning-of-sequence markers and bias matrices omitted in the attention walkthrough 5. Positional embeddings (RoPE) simplified; see the original papers