16. CSA/HCA: Compressed Self-Attention / Hybrid Attention (DeepSeek-V4)
Source: DeepSeek-V4-Pro technical report (HuggingFace)
Core question: As DeepSeek's next-generation architecture, how does V4 evolve beyond V3's MLA + DSA? Can the attention module itself become more compressed and more hybrid?
Method innovations
CSA (Compressed Self-Attention) and HCA (Hybrid Attention) are new components introduced in DeepSeek-V4. The technical report offers limited detail; the known information is:
1. CSA (Compressed Self-Attention): Further compresses attention computation and storage on top of MLA. This may involve more aggressive latent-space compression or dynamic compression strategies.
2. HCA (Hybrid Attention): Uses different types of attention mechanisms mixed within the same model. This may include:
- Mixing local attention with global attention
- Mixing standard attention with linear attention
- Mixing attention layers with different compression ratios
- DeepSeek-V4-Pro is DeepSeek's next-generation flagship model
- CSA/HCA is one of its core architectural innovations
- Specific implementation details await the formal paper release
- Shazeer (2019). Fast Transformer Decoding: One Write-Head is All You Need. arXiv:1911.02150
- Ainslie et al. (2023). GQA: Training Generalized Multi-Query Transformer Models. arXiv:2305.13245
- DeepSeek-AI (2024). DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model. arXiv:2405.04434
- Child et al. (2019). Generating Long Sequences with Sparse Transformers. arXiv:1904.10509
- Beltagy et al. (2020). Longformer: The Long-Document Transformer. arXiv:2004.05150
- Gemma Team (2024). Gemma 2: Improving Open Language Models at a Practical Size. arXiv:2408.00118
- DeepSeek-AI (2025). DeepSeek-V3.2 Technical Report. arXiv:2512.02556
- DeepSeek-AI (2026). DeepSeek-V4-Pro Technical Report. HuggingFace
Key information
Impact assessment
CSA/HCA represents the "hybridization" trend in attention architecture—not choosing one attention mechanism, but letting the model use the most suitable attention in different layers/scenarios. This aligns with trends such as Gemma 2's "interleaved local-global" attention and Kimi Linear's "hybrid KDA+MLA" design.
Feynman-style commentary
> CSA/HCA can be summarized as "no silver bullet." For short sequences, full attention works best; for long sequences, sparse attention is better; for some tasks, linear attention is sufficient. Rather than debating which attention "wins," let the model itself decide when to use what. It's like a good toolbox—not just a hammer, but a hammer, screwdriver, and wrench, used as needed. Feynman would say: don't ask "which theory is correct," ask "under which conditions is which theory useful."
---