CSA/HCA: Compressed Self-Attention and Hybrid Attention in DeepSeek-V4

16. CSA/HCA: Compressed Self-Attention / Hybrid Attention (DeepSeek-V4)

Source: DeepSeek-V4-Pro Technical Report (HuggingFace)

Core question: As DeepSeek's next-generation architecture, how does DeepSeek-V4 evolve beyond V3's MLA + DSA? Can the attention module itself be further compressed and hybridized?

Method innovations: CSA (Compressed Self-Attention) and HCA (Hybrid Attention) are new components introduced in DeepSeek-V4. Details in the technical report are limited; the known information is as follows:

1. CSA (Compressed Self-Attention): Further compresses attention computation and storage on top of MLA. This may involve more aggressive latent-space compression, or dynamic compression strategies.

2. HCA (Hybrid Attention): Mixing different types of attention mechanisms within the same model. This may include:

• Mixtures of local attention and global attention
• Mixtures of standard attention and linear attention
• Attention layers with different compression ratios

Key information:

• DeepSeek-V4-Pro is DeepSeek's next-generation flagship model
• CSA/HCA is one of its core architectural innovations
• Specific implementation details await the official paper release

Impact assessment: CSA/HCA represents the "hybridization" trend in attention architectures—not choosing a single attention mechanism, but letting the model use the most suitable attention for different layers and scenarios. This aligns with trends such as Gemma 2's interleaved local-global attention and Kimi Linear's hybrid KDA+MLA.

Feynman-style commentary: > The idea behind CSA/HCA can be summarized as "no silver bullet." For short sequences, full attention is best; for long sequences, sparse attention is better; for certain tasks, linear attention is sufficient. Rather than arguing over which attention "wins," let the model itself decide when to use what. It's like a good toolbox—not just a hammer, but a hammer, screwdriver, and wrench, taken as needed. Feynman would say: don't ask "which theory is right," ask "under what conditions is which theory useful."

---

References:

• Shazeer (2019). Fast Transformer Decoding: One Write-Head is All You Need. arXiv:1911.02150
• Ainslie et al. (2023). GQA: Training Generalized Multi-Query Transformer Models. arXiv:2305.13245
• DeepSeek-AI (2024). DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model. arXiv:2405.04434
• Child et al. (2019). Generating Long Sequences with Sparse Transformers. arXiv:1904.10509
• Beltagy et al. (2020). Longformer: The Long-Document Transformer. arXiv:2004.05150
• Gemma Team (2024). Gemma 2: Improving Open Language Models at a Practical Size. arXiv:2408.00118
• DeepSeek-AI (2025). DeepSeek-V3.2 Technical Report. arXiv:2512.02556
• DeepSeek-AI (2026). DeepSeek-V4-Pro Technical Report. HuggingFace