DeepSeek-V4: Native Million-Token Context with Hybrid Compressed Attention
This post presents a detailed walkthrough of the DeepSeek-V4 technical report, introducing the DeepSeek-V4-Pro and DeepSeek-V4-Flash models with native one-million-token context support.
Key points
- Model family: DeepSeek-V4-Pro (1.6T total parameters, 49B activated) and DeepSeek-V4-Flash (284B total, 13B activated), both natively supporting 1M-token contexts.
- Inherited foundations: The series retains DeepSeek-V3's core framework — DeepSeekMoE (sparse expert activation) and Multi-Token Prediction (MTP) — providing a stable base for the new innovations.
- mHC (Manifold-Constrained Hyper-Connections): An upgrade over plain residual connections. Connectivity weights are constrained via doubly stochastic matrices with Sinkhorn-Knopp projection, giving smoother information flow and more stable training over very long sequences.
- Hybrid attention — CSA + HCA: Compressed Sparse Attention compresses the KV cache along the sequence dimension before applying DeepSeek Sparse Attention (DSA); Heavily Compressed Attention applies stronger KV compression while keeping dense attention. This removes the quadratic bottleneck at long context.
- Efficiency at 1M tokens (reported figures): V4-Pro needs only 27% of V3.2's single-token inference FLOPs (FP8-equivalent) with a 10% KV cache; V4-Flash drops to 10% FLOPs and 7% KV cache.
- Muon optimizer: Uses Hybrid Newton-Schulz iteration for faster, more stable convergence; pretraining runs on 32T+ high-quality, diverse tokens.
- Infrastructure: Fine-grained communication-computation overlapping for expert parallelism, the TileLang DSL, batch-invariant deterministic kernels, FP4 quantization-aware training (MoE expert weights and QK paths), context parallelism for compressed attention, and heterogeneous on-disk KV caching for shared-prefix reuse at inference.
Reported benchmarks (V4-Pro-Max)
| Category | Results | |----------|---------| | Knowledge / reasoning | MMLU-Pro 91.0%, GPQA 94.3%, HLE 44.4% | | Code | LiveCodeBench 93.5%, Codeforces 3206, SWE-Verified 80.8% | | Long context | MRCR @1M 92.9% | | Real-world tasks | Chinese writing, search, white-collar tasks, code agents |
Post-training
Post-training includes domain-specific Specialist Training merged via On-Policy Distillation (OPD), plus RL infrastructure featuring FP4 quantization integration, hidden-state caching with asynchronous teacher scheduling, preemptible fault-tolerant rollout services, million-token-context RL scaling, and agentic AI sandboxes.
Thinking management
V4 adds fast-command special tokens (e.g., <|action|>, <|query|>) and optimized interleaved thinking: reasoning traces are preserved throughout in tool-use scenarios and selectively cleaned in ordinary conversation, enabling cross-turn continuity.
Conclusion
The article concludes that DeepSeek-V4's efficient million-token context opens a new stage for test-time scaling and long-horizon agentic tasks, positioning the series as a foundation for future paradigms such as online learning and agentic AI, while acknowledging remaining optimization opportunities.
References
1. DeepSeek-AI. DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence. 2026. https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro/blob/main/DeepSeek_V4.pdf 2. DeepSeek-AI. DeepSeek-V3 Technical Report. 2024. 3. DeepSeek-AI. DeepSeek-V3.2 Post-Training Enhancements. 2025.