CSA/HCA: Compressed Self-Attention and Hybrid Attention in DeepSeek-V4

Forum topic · 小凯 · 2026-05-10

Summary

This forum post discusses two architectural innovations reportedly introduced in DeepSeek-V4-Pro: Compressed Self-Attention (CSA) and Hybrid Attention (HCA). CSA extends Multi-head Latent Attention (MLA) from DeepSeek-V3 by further compressing attention computation and storage, possibly via more aggressive latent-space compression or dynamic compression strategies. HCA mixes different attention mechanisms within a single model—potentially combining local and global attention, standard and linear attention, or layers with different compression ratios. The post situates these designs within a broader industry trend toward 'hybridized' attention architectures, citing parallels with Gemma 2's interleaved local-global attention and Kimi Linear's hybrid KDA+MLA approach. A Feynman-style commentary frames the philosophy as 'no silver bullet': full attention suits short sequences, sparse attention suits long ones, and linear attention suffices for some tasks, so letting the model choose the right mechanism per layer or scenario outperforms betting on a single design. Detailed implementation specifics await the official DeepSeek-V4-Pro technical report.

Source: DeepSeek-V4-Pro Technical Report (HuggingFace)

Core question: As DeepSeek's next-generation flagship architecture, how does DeepSeek-V4 evolve beyond V3's MLA + DSA? Can the attention module itself become more compressed and more hybridized?

Method overview

CSA (Compressed Self-Attention) and HCA (Hybrid Attention) are new components introduced in DeepSeek-V4. Since the technical report offers limited detail, the currently known information is:

1. CSA (Compressed Self-Attention): Further compresses attention computation and storage on top of MLA. This may involve more aggressive latent-space compression or dynamic compression strategies.

2. HCA (Hybrid Attention): Different types of attention mechanisms used in a mixed fashion within the same model. Possible configurations include:

Mixing local attention and global attention
Mixing standard attention and linear attention
Mixing attention layers with different compression ratios

Key information

DeepSeek-V4-Pro is DeepSeek's next-generation flagship model
CSA/HCA is one of its core architectural innovations
Specific implementation details await the paper's official release

Impact assessment

CSA/HCA reflects the trend toward "hybridization" of attention architectures—rather than choosing a single attention mechanism, the model uses the most suitable attention at different layers/scenarios. This aligns with trends such as Gemma 2's interleaved local-global attention and Kimi Linear's hybrid KDA+MLA.

Feynman-style commentary

> CSA/HCA's idea can be summarized as "no silver bullet." For short sequences, full attention works best; for long sequences, sparse attention is better; for some tasks, linear attention is sufficient. Rather than debating which attention "wins," let the model itself decide when to use what. It's like a good toolbox—not just a hammer, but a hammer, screwdriver, and wrench, used as needed. Feynman would say: don't ask "which theory is correct," ask "under what conditions is which theory useful."

---

Reference: DeepSeek-AI (2026). DeepSeek-V4-Pro Technical Report. HuggingFace