DeepSeek-V4 Explained: Efficient Million-Token Context via Hybrid Compressed Attention

Forum topic · ✨步子哥 · 2026-04-24

Summary

This post analyzes the DeepSeek-V4 technical report, covering the new model family: DeepSeek-V4-Pro (1.6T total / 49B active parameters) and DeepSeek-V4-Flash (284B total / 13B active), both natively supporting one-million-token contexts. Building on DeepSeek-V3's DeepSeekMoE architecture and Multi-Token Prediction, V4 introduces Manifold-Constrained Hyper-Connections (mHC) for improved residual information flow, and a hybrid attention scheme combining Compressed Sparse Attention (CSA) and Heavily Compressed Attention (HCA). At 1M tokens, V4-Pro reportedly needs only 27% of V3.2's single-token inference FLOPs with 10% of the KV cache; Flash reaches 10% FLOPs and 7% KV cache. Training uses the Muon optimizer, FP4 quantization-aware training, TileLang-based kernels, and 32T+ tokens of pretraining data. Reported results include MMLU-Pro 91.0%, GPQA 94.3%, HLE 44.4%, LiveCodeBench 93.5%, Codeforces 3206, MRCR@1M 92.9%, and SWE-Verified 80.8%. Post-training covers Specialist Training, On-Policy Distillation, and RL infrastructure with million-token context scaling.

DeepSeek-V4: Efficient Million-Token Context Intelligence

This is an English translation/summary of a Chinese forum post discussing the DeepSeek-V4 technical report (referenced as DeepSeek_V4.pdf).

Key points

Model family: DeepSeek-V4-Pro (1.6T total / 49B active parameters) and DeepSeek-V4-Flash (284B total / 13B active), both natively supporting 1,000,000-token contexts, addressing the quadratic complexity bottleneck of attention that limits test-time scaling and long-horizon agentic tasks.
Inherited foundation: V4 retains DeepSeek-V3's core design — DeepSeekMoE (sparse expert activation) and Multi-Token Prediction (MTP).
mHC (Manifold-Constrained Hyper-Connections): an upgraded residual connection scheme constrained via doubly stochastic matrices and Sinkhorn-Knott projection, improving information fidelity and training stability over long sequences.
Hybrid attention: combines Compressed Sparse Attention (CSA) — KV compression along the sequence dimension followed by DeepSeek Sparse Attention (DSA) — and Heavily Compressed Attention (HCA) — aggressive KV compression with dense attention. At 1M-token context:
V4-Pro: 27% of V3.2's single-token inference FLOPs (FP8-equivalent), 10% KV cache.
V4-Flash: 10% FLOPs, 7% KV cache.
Muon optimizer: uses Hybrid Newton-Schulz iteration for faster, more stable convergence; pretraining ran on 32T+ high-quality, diverse tokens.
Infrastructure: fine-grained communication-computation overlap for expert parallelism, TileLang DSL, batch-invariant deterministic kernels, FP4 quantization-aware training (MoE expert weights and QK paths), context parallelism for compressed attention, heterogeneous KV cache with on-disk storage for shared-prefix reuse.

Benchmark results (V4-Pro-Max, reported)

| Category | Results | |----------|---------| | Knowledge / reasoning | MMLU-Pro 91.0%, GPQA 94.3%, HLE 44.4% | | Code | LiveCodeBench 93.5%, Codeforces 3206, SWE-Verified 80.8% | | Long context | MRCR @1M: 92.9% | | Real-world tasks | Chinese writing, search, white-collar tasks, code agents |

Post-training

Domain-specific Specialist Training merged via On-Policy Distillation (OPD).
RL infrastructure: FP4 quantization, teacher scheduling with hidden-state caching and async loading, preemptible fault-tolerant rollout services, million-token-context RL scaling, agentic AI sandboxes.
Fast-instruction special tokens (e.g. <|action|>, <|query|>) and improved interleaved thinking: reasoning traces preserved across tool-use turns, selectively cleaned in ordinary chat.

Conclusion

The report positions V4 as a foundation for test-time scaling, online learning, and long-horizon agentic AI, while acknowledging remaining optimization opportunities in specific scenarios.