How DeepSeek Rewrote the Transformer: MLA's 57x Compression Magic

How DeepSeek Rewrote the Transformer: MLA's 57x Compression Magic

> Original video: How DeepSeek Rewrote the Transformer [MLA] by Welch Labs > Video: http://www.youtube.com/watch?v=kYI-U1_I3mQ > Papers: DeepSeek-V2 (arXiv:2405.04434), DeepSeek-V3 (arXiv:2412.19437), DeepSeek-R1 (arXiv:2501.12948)

The 4MB-per-token disaster

Every time a large language model generates a token, it stores a "memory" in GPU VRAM: the Key and Value vectors of attention (the KV cache). For a model at the scale of DeepSeek-R1 using traditional multi-head attention (MHA), the KV cache consumes roughly 4MB of VRAM per token.

4MB sounds small, but with a 10K-token context that's 40GB — beyond most consumer GPUs. At 100K tokens: 400GB. This is the fatal bottleneck of the classic Transformer: the KV cache grows linearly with both context length and model size — a double explosion.

DeepSeek's answer is MLA (Multi-head Latent Attention). Rather than buying bigger GPUs, it improves the Transformer's memory efficiency by ~57x (estimated for V3/R1).

Why the KV cache explodes: starting from MHA

In each Transformer layer, the model has multiple attention heads (e.g., 64), each maintaining its own Key and Value vectors. With \(L\) layers, \(h\) heads, head dimension \(d_k\), and sequence length \(N\):

For DeepSeek-R1 (671B parameters, 37B active), standard MHA would require roughly 3,997,696 parameters per token in the KV cache.

The research community's earlier fixes were to cut heads:

• MQA (Multi-Query Attention): all Query heads share 1 KV head; cache shrinks to \(1/h\).
• GQA (Grouped-Query Attention): Query heads share KV heads in groups; cache shrinks to \(h_{group}/h\).

These reduce memory but degrade quality — shared KV heads can't capture the diverse information different Query heads need. DeepSeek's take: don't cut heads, compress.

MLA's core insight: low-rank compression

Instead of storing full K/V vectors, MLA stores a highly compressed latent vector, decompressing on demand:

• MHA stores every frame of a movie uncompressed
• MQA/GQA makes fewer copies at native resolution
• MLA compresses the movie to MP4 and decodes in real time during playback

Mathematically, for each token MLA stores:

\[c_t^{KV} = [W^{DK} K_t; W^{DV} V_t] \in \mathbb{R}^{d_c}\]

where \(d_c \ll d_k\) (e.g., \(d_c = 512\) vs \(d_k = 4096\)). Up-projection matrices restore the multi-head KV when needed:

\[\tilde{K}_t = W^{UK} c_t^{KV}, \quad \tilde{V}_t = W^{UV} c_t^{KV}\]

The DeepSeek-V2 paper claims a 93.3% KV cache reduction; Welch Labs recomputes from V3/R1 architecture parameters an estimated 56.9x reduction (from 3,997,696 to 70,272 parameters per token). Either way: 128K context comfortably fits on ordinary GPUs.

Absorbed projections: the mathematician's shortcut

The naive MLA pipeline decompresses \(c^{KV}\) via \(W^{UK}\), computes attention, then projects output via \(W^O\). Since these are fixed weight matrices, matrix associativity lets you absorb them:

1. Query-side absorption: \(W^{UQ}\) merges with the \(W^{UK}\) path. 2. Output-side absorption: \(W^{UV}\) and \(W^O\) merge into one matrix. 3. V reused from K: Value vectors aren't stored separately; they're taken from the first \(d_c\) dimensions of the Key path.

The key point: this is not an approximation. By reordering multiplications, the model skips explicit decompression at inference — the results are mathematically identical, but compute drops substantially.

Dancing with RoPE: decoupled positional encoding

MLA has a hidden problem: RoPE injects position into Q and K, but MLA computes them in a low-dimensional latent space. Applying RoPE directly to the low-dim vectors would distort positional information after decompression.

DeepSeek's solution is Decoupled RoPE: position-carrying Q/K dimensions keep a small independent vector with positional encoding; everything else goes through the low-rank compressed path; the two are concatenated. Positional information flows through a separate "express lane," undisturbed by compression.

Why DeepSeek's API is so cheap

1. Fewer GPUs serve bigger models — tiny KV cache means more context per GPU. 2. Faster inference — the DeepSeek-V2 paper reports just under 6x throughput improvement from MLA; possibly more for V3/R1. 3. Memory bandwidth stops being the bottleneck — Transformer inference is usually memory-bandwidth-bound reading the KV cache; compressing it 57x lets GPU compute units actually work.

Evolution of attention: MHA → MQA/GQA → MLA

| Mechanism | Idea | KV cache | Quality cost | Models | |:---|:---|:---|:---|:---| | MHA | Each Query head owns its KV head | \(2 \times h \times d_k\) | Baseline | GPT-3, LLaMA-1 | | MQA | All Query heads share 1 KV head | \(2 \times d_k\) | Significant drop | PaLM, Falcon | | GQA | Grouped sharing | \(2 \times g \times d_k\) | Moderate drop | LLaMA-2, Mistral | | MLA | Low-rank compression + absorbed projections | \(2 \times d_c\) (\(d_c \ll d_k\)) | Nearly none | DeepSeek-V2/3/R1 |

MLA's breakthrough: it trades memory for compression, not fewer heads — retaining multi-head expressiveness while approaching MQA-level storage.

Hardware perspective

A 2025 hardware analysis (Robin Geens, KU Leuven) notes that MLA doesn't just reduce bandwidth — it can shift attention from memory-bound to compute-bound. Future accelerators optimized for MLA could outperform MHA-optimized ones by an order of magnitude; MLA is a software-hardware co-design opportunity.

Limitations and open questions

1. Higher training cost: compression matrices must be learned during pretraining (though inference is cheap). 2. Ecosystem compatibility: MLA requires bespoke support in HuggingFace, vLLM, etc.; kernels like FlashMLA were written by DeepSeek itself. 3. Attention-pattern flexibility: whether fixed compression matrices limit certain attention patterns remains unsettled. 4. Ultra-long context (>128K): whether compression errors accumulate at 1M+ tokens needs more research.

Closing thoughts

DeepSeek's rewrite is a philosophical shift in how memory is stored: MHA stores memory verbatim; MQA/GQA shares it at a loss; MLA says memory can be compressed and restored exactly at compute time. The 57x compression is math — low-rank projection, matrix absorption, decouped RoPE — not magic.

If an LLM's memory can compress 57x without accuracy loss, what's the human brain's compression ratio? We remember compressed "semantic vectors" and decompress details on demand. MLA may be our first look at a genuine compressed-memory mechanism in artificial neural networks. The next step: compressing entire weight layers — or all of a model's knowledge — into latent vectors that decompress on demand.

References

• Video: How DeepSeek Rewrote the Transformer [MLA] by Welch Labs — http://www.youtube.com/watch?v=kYI-U1_I3mQ
• Papers:
• DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model (arXiv:2405.04434)
• DeepSeek-V3 (arXiv:2412.19437)
• DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning (arXiv:2501.12948)
• Hardware analysis: Hardware-Centric Analysis of DeepSeek's Multi-Head Latent Attention (Robin Geens, KU Leuven, 2025)
• Migration method: MHA2MLA: Towards Economical Inference (arXiv:2502.14837) — adapts MHA models like Llama to MLA with only 0.3%–0.6% of data, reducing KV cache by 92.19%
• Video author's notes: (1) V2 paper claims 93.3% KV cache reduction; (2) V3/R1 estimated at 56.9x (3,997,696/70,272); (3) throughput gains just under 6x for V2, possibly more for V3/R1; (4) start-of-sequence tokens and bias matrices omitted from the attention derivation; (5) positional embedding (RoPE) simplified in the video — see the original papers.