DSA: DeepSeek Sparse Attention in DeepSeek-V3.2 (2025)

Forum topic · 小凯 · 2026-05-10

Summary

DSA (DeepSeek Sparse Attention) is a core architectural innovation in DeepSeek-V3.2 (arXiv 2512.02556), designed to tackle the O(n²) complexity of attention as context lengths grow to 128K and beyond. Unlike earlier sparse approaches such as sliding-window attention or Sparse Transformer, which either lose too much information or are complex to implement, DSA sharply reduces long-context compute while preserving model quality. It combines with MLA (Multi-head Latent Attention) so that compute is reduced further after the KV cache is already compressed, and it ships with hardware-aware CUDA kernels so the sparsity is fast on real GPUs, not just in theory. Reported results include DeepSeek-V3.2 performing comparably to GPT-5 on reasoning and agent tasks, the high-compute V3.2-Speciale variant achieving gold-medal-level results at the 2025 IMO and IOI, and 96% vote-coverage retrieval capability (SIRA component). DSA marks the maturation of sparse attention from research concept to production deployment, suggesting future LLMs will win through smarter sparsity rather than sheer scale.

15. DSA: DeepSeek Sparse Attention (2025, DeepSeek-AI)

arXiv: 2512.02556 (DeepSeek-V3.2 technical report)

Core problem: As context lengths grow to 128K, 1M tokens, and beyond, attention's O(n²) complexity once again becomes the bottleneck. Existing sparse approaches (SWA, Sparse Transformer) either lose too much information or are complex to implement. How can DeepSeek drastically reduce long-context compute cost while preserving model quality?

Method innovation: DSA (DeepSeek Sparse Attention) is one of the core architectural innovations in DeepSeek-V3.2. Although paper details are limited (from the V3.2 technical report), what is known:

1. Sparse attention mechanism: Significantly reduces computational complexity in long-context scenarios while maintaining model performance 2. Combined with MLA: On top of the already-compressed KV cache, sparsification further reduces compute 3. Hardware-aware implementation: Optimized CUDA kernels ensure sparse attention is actually fast on real GPUs (not just theoretically fast)

Key numbers:

DeepSeek-V3.2 is "comparable to GPT-5" on reasoning and agent tasks
The high-compute variant DeepSeek-V3.2-Speciale achieves gold-medal-level performance at the 2025 IMO and IOI
96% vote-coverage retrieval capability (SIRA component)

Impact assessment: DSA represents the maturation of sparse attention from "research toy" to "production deployment". DeepSeek-V3.2 maintains top-tier performance while achieving efficient inference through the DSA + MLA combination. This shows that the LLMs of the future are not about "bigger is better", but "smarter sparsity is better".

Feynman-style commentary: > The concrete implementation details of DSA are not fully public yet, but the direction is clear: the next battlefield for attention is "how to cleverly not compute". Not all token pairs need attention; not all layers need a global view. DSA's mindset is "structural sparsity" — dynamically selecting the attention scope based on data characteristics and task requirements. This reminds me of Feynman on quantum mechanics: "Nature does not run according to your mathematical convenience." Good sparse schemes are not designed for mathematical convenience; they are designed according to data structure and hardware reality.

---

arXiv: 2512.02556 (DeepSeek-V3.2 technical report)

DSA: DeepSeek Sparse Attention in DeepSeek-V3.2 (2025)

Summary

15. DSA: DeepSeek Sparse Attention (2025, DeepSeek-AI)

Tags