CSA/HCA: Compressed Self-Attention and Hybrid Attention in DeepSeek-V4

Forum topic · 小凯 · 2026-05-10

Summary

This post analyzes CSA (Compressed Self-Attention) and HCA (Hybrid Attention), the attention innovations attributed to DeepSeek-V4-Pro. CSA reportedly builds on Multi-head Latent Attention (MLA) by further compressing attention computation and storage, possibly through more aggressive latent-space compression or dynamic compression strategies. HCA mixes different attention types within one model — local and global attention, standard and linear attention, or layers with varying compression ratios. Official technical details remain limited pending the full paper, but the post frames this as part of a broader 'hybridization' trend in attention design, alongside Gemma 2's interleaved local-global attention and Kimi Linear's mixed KDA+MLA approach. The author's commentary argues there is no single silver bullet: full attention suits short sequences, sparse attention long ones, and linear attention suffices for some tasks — so letting the model select the right mechanism per layer is like owning a full toolbox. The article includes references to MLA/GQA lineage, sparse attention work, and the DeepSeek-V2/V3.2/V4-Pro reports.

16. CSA/HCA: Compressed Self-Attention / Hybrid Attention (DeepSeek-V4)

Source: DeepSeek-V4-Pro Technical Report (HuggingFace)

Core question: As DeepSeek's next-generation architecture, how does DeepSeek-V4 evolve beyond V3's MLA + DSA? Can the attention module itself become more compressed and more hybrid?

Methodological innovations

CSA (Compressed Self-Attention) and HCA (Hybrid Attention) are new components introduced in DeepSeek-V4. As the technical report provides limited detail, what is currently known:

1. CSA (Compressed Self-Attention): Further compresses attention computation and storage on top of MLA. This may involve more aggressive latent-space compression or dynamic compression strategies.

2. HCA (Hybrid Attention): Different types of attention mechanisms mixed within the same model. Possibilities include:

Mixing local attention and global attention
Mixing standard attention and linear attention
Mixing attention layers with different compression ratios

Key information

DeepSeek-V4-Pro is DeepSeek's next-generation flagship model
CSA/HCA is one of its core architectural innovations
Concrete implementation details await the official paper release

Impact assessment

CSA/HCA represents the "hybridization" trend in attention architecture — rather than choosing a single attention mechanism, the model uses the most suitable attention for different layers and scenarios. This aligns with trends such as Gemma 2's "interleaved local-global" attention and Kimi Linear's "hybrid KDA+MLA."

Feynman-style commentary

> The idea behind CSA/HCA can be summarized as "no silver bullet." For short sequences, full attention is best; for long sequences, sparse attention is better; for some tasks, linear attention is sufficient. Rather than arguing over which attention "wins," let the model decide when to use what. It's like a good toolbox — not just a hammer, but a hammer, screwdriver, and wrench, used as needed. Feynman would say: don't ask "which theory is correct," ask "under what conditions is each theory useful."

References

Shazeer (2019). Fast Transformer Decoding: One Write-Head is All You Need. arXiv:1911.02150
Ainslie et al. (2023). GQA: Training Generalized Multi-Query Transformer Models. arXiv:2305.13245
DeepSeek-AI (2024). DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model. arXiv:2405.04434
Child et al. (2019). Generating Long Sequences with Sparse Transformers. arXiv:1904.10509
Beltagy et al. (2020). Longformer: The Long-Document Transformer. arXiv:2004.05150
Gemma Team (2024). Gemma 2: Improving Open Language Models at a Practical Size. arXiv:2408.00118
DeepSeek-AI (2025). DeepSeek-V3.2 Technical Report. arXiv:2512.02556
DeepSeek-AI (2026). DeepSeek-V4-Pro Technical Report. HuggingFace