16. CSA/HCA: Compressed Self-Attention / Hybrid Attention (DeepSeek-V4)
Source: DeepSeek-V4-Pro Technical Report (HuggingFace)
Core question: As a next-generation architecture, how does DeepSeek-V4 evolve beyond the MLA + DSA foundation of V3? Can the attention module itself be made more compressed and more hybrid?
Methodological innovations
CSA (Compressed Self-Attention) and HCA (Hybrid Attention) are new components introduced in DeepSeek-V4. Since the technical report's details are limited, the known information is:
1. CSA (Compressed Self-Attention): Further compresses attention computation and storage on top of MLA. This may involve more aggressive latent-space compression, or dynamic compression strategies.
2. HCA (Hybrid Attention): Mixing different types of attention mechanisms within a single model. This may include:
- Hybrid local attention and global attention
- Hybrid standard attention and linear attention
- Mixing attention layers with different compression ratios
- DeepSeek-V4-Pro is DeepSeek's next-generation flagship model
- CSA/HCA is one of its core architectural innovations
- Specific implementation details await the official paper release
- Shazeer (2019). Fast Transformer Decoding: One Write-Head is All You Need. arXiv:1911.02150
- Ainslie et al. (2023). GQA: Training Generalized Multi-Query Transformer Models. arXiv:2305.13245
- DeepSeek-AI (2024). DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model. arXiv:2405.04434
- Child et al. (2019). Generating Long Sequences with Sparse Transformers. arXiv:1904.10509
- Beltagy et al. (2020). Longformer: The Long-Document Transformer. arXiv:2004.05150
- Gemma Team (2024). Gemma 2: Improving Open Language Models at a Practical Size. arXiv:2408.00118
- DeepSeek-AI (2025). DeepSeek-V3.2 Technical Report. arXiv:2512.02556
- DeepSeek-AI (2026). DeepSeek-V4-Pro Technical Report. HuggingFace
Key information
Impact assessment
CSA/HCA represents the "hybridization" trend in attention architectures — rather than choosing a single attention mechanism, the model uses the most suitable attention at different layers and in different scenarios. This aligns with trends such as Gemma 2's "interleaved local-global" attention and Kimi Linear's "hybrid KDA + MLA."
Feynman-style commentary
> The idea behind CSA/HCA can be summarized as "there is no silver bullet." For short sequences, full attention is best; for long sequences, sparse attention is better; for some tasks, linear attention is sufficient. Rather than debating which attention mechanism "wins," it is better to let the model decide when to use which. It is like a good toolbox — not just a hammer, but a hammer, screwdriver, and wrench, picked as needed. Feynman would say: don't ask "which theory is correct," ask "under what conditions is each theory useful."
---