CSA/HCA: Compressed Self-Attention / Hybrid Attention (DeepSeek-V4)
Source: DeepSeek-V4-Pro Technical Report (HuggingFace)
Core question: As DeepSeek's next-generation architecture, how does DeepSeek-V4 evolve beyond V3's MLA + DSA? Can the attention module itself become more compressed and more hybrid?
Method overview
CSA (Compressed Self-Attention) and HCA (Hybrid Attention) are new components introduced in DeepSeek-V4. Since the technical report offers limited detail, the currently known information is:
1. CSA (Compressed Self-Attention): Further compresses the computation and storage of attention on top of MLA. This may involve more aggressive latent-space compression or dynamic compression strategies.
2. HCA (Hybrid Attention): Mixing different types of attention mechanisms within a single model. This may include:
- Mixing local attention and global attention
- Mixing standard attention and linear attention
- Mixing attention layers with different compression ratios
- DeepSeek-V4-Pro is DeepSeek's next-generation flagship model
- CSA/HCA is one of its core architectural innovations
- Concrete implementation details await the formal paper release
- Shazeer (2019). Fast Transformer Decoding: One Write-Head is All You Need. arXiv:1911.02150
- Ainslie et al. (2023). GQA: Training Generalized Multi-Query Transformer Models. arXiv:2305.13245
- DeepSeek-AI (2024). DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model. arXiv:2405.04434
- Child et al. (2019). Generating Long Sequences with Sparse Transformers. arXiv:1904.10509
- Beltagy et al. (2020). Longformer: The Long-Document Transformer. arXiv:2004.05150
- Gemma Team (2024). Gemma 2: Improving Open Language Models at a Practical Size. arXiv:2408.00118
- DeepSeek-AI (2025). DeepSeek-V3.2 Technical Report. arXiv:2512.02556
- DeepSeek-AI (2026). DeepSeek-V4-Pro Technical Report. HuggingFace
Key information
Impact assessment
CSA/HCA represents the trend toward "hybridization" of attention architectures—rather than picking a single attention mechanism, the model uses the most suitable attention at different layers and in different scenarios. This aligns with trends such as Gemma 2's interleaved local-global attention and Kimi Linear's hybrid KDA+MLA approach.
Commentary
> CSA/HCA's idea can be summarized as "no silver bullet." For short sequences, full attention is best; for long sequences, sparse attention is better; for some tasks, linear attention is sufficient. Rather than arguing over which attention mechanism "won," let the model decide when to use what. It's like a good toolbox—not just a hammer, but a hammer, screwdriver, and wrench, used as needed. Feynman would say: don't ask "which theory is correct," ask "under what conditions is which theory useful."
---