Mamba-2: State Space Duality (2024, Gu & Dao)
arXiv: 2405.21060
Core Problem
Mamba is an SSM; Transformers use attention — two seemingly very different architectures. Yet both do the same thing: sequence modeling. Is there a unified mathematical framework connecting the two? And if so, can it be used to design faster, better architectures?
Methodological Innovation
The paper's central finding is State Space Duality (SSD): SSMs and attention are essentially different perspectives on the same mathematical object.
Specifically:
- SSM recurrent view:
h_t = f(h_{t-1}, x_t)— compress history into a state - Attention global view:
y_t = Σ_i attention(t, i) · x_i— directly query history - Core layer speedup: 2-8x
- Language modeling performance on par with Transformers
- SSD framework unifies the theory of SSMs and attention
The SSD framework proves that for a special class of structured semiseparable matrices, these two views can be converted into each other. Mamba's selective SSM corresponds to a special form of attention; conversely, standard attention can be seen as a special kind of SSM.
Based on this theory, Mamba-2's core layer is a refined version of Mamba's selective SSM:
1. Leverages SSD duality to design a new computation path 2. The core layer is 2-8x faster than Mamba 3. Remains competitive with Transformers
Key Numbers
Impact Assessment
Mamba-2's SSD framework is a theoretical milestone. It shows that attention and SSMs are not "competitors" but "two sides of the same coin." This unified perspective inspired subsequent hybrid architectures (e.g., KDA's layer-level mixing, Gated DeltaNet's hybrid layers).
Feynman-style Commentary
> Mamba-2's real value is finding a "unified grammar." Before, attention and SSMs were two camps, each with its own papers, conferences, and fan bases. SSD says: you're all speaking the same language, just different dialects. This unification is not philosophical — it's mathematical (structured semiseparable matrices). Once the grammar is unified, you can design models with "mixed dialects" — some layers speaking the attention dialect, others the SSM dialect. Feynman would say: when you discover that two seemingly different things are actually the same thing, you've reached a higher level of understanding.
arXiv: 2405.21060