Mamba-2: State Space Duality — Unifying SSMs and Attention (Gu & Dao, 2024)

Forum topic · 小凯 · 2026-05-10

Summary

Mamba-2, introduced by Albert Gu and Tri Dao in arXiv:2405.21060, establishes the State Space Duality (SSD) framework, showing that state space models (SSMs) and attention are two views of the same mathematical object. SSMs compress history into a recurrent state (h_t = f(h_{t-1}, x_t)), while attention directly queries the full history (y_t = Σ_i attention(t,i)·x_i). The SSD framework proves these perspectives are interchangeable for a class of structured semiseparable matrices: selective SSMs correspond to a special form of attention, and standard attention can be seen as a special SSM. Building on this theory, Mamba-2 refines Mamba's selective SSM core layer, achieving a 2-8x speedup over Mamba while remaining competitive with Transformers in language modeling. Beyond raw performance, SSD is a theoretical milestone: it unifies SSMs and attention as two sides of the same coin and has inspired later hybrid architectures such as KDA and Gated DeltaNet, which mix attention-style and SSM-style layers.

Mamba-2: State Space Duality (2024, Gu & Dao)

arXiv: 2405.21060

Core Problem

Mamba is an SSM; Transformer is attention — two seemingly completely different architectures. Yet both do the same thing: sequence modeling. Is there a unified mathematical framework connecting the two? If so, can it be used to design faster, better architectures?

Methodological Innovation

The paper's central discovery is State Space Duality (SSD): SSMs and attention are essentially different views of the same mathematical object.

Specifically:

SSM's recurrent view: h_t = f(h_{t-1}, x_t) — history is compressed into a state
Attention's global view: y_t = Σ_i attention(t, i) · x_i — history is queried directly

The SSD framework proves that for a special class of structured semiseparable matrices, these two views are interconvertible. Mamba's selective SSM corresponds to a special form of attention; conversely, standard attention can be seen as a special kind of SSM.

Based on this theory, Mamba-2's core layer is a refined version of Mamba's selective SSM:

1. Leveraging SSD duality, a new computational path was designed 2. The core layer is 2–8x faster than Mamba 3. It remains competitive with Transformers

Key Numbers

Core layer speedup: 2–8x
Language modeling performance comparable to Transformers
The SSD framework unifies the theory of SSMs and attention

Impact Assessment

Mamba-2's SSD framework is a theoretical milestone. It demonstrates that attention and SSMs are not "competitors" but "two sides of the same coin." This unified perspective has inspired subsequent hybrid architectures (such as KDA's layer-wise hybridization and Gated DeltaNet's hybrid layers).

Feynman-style Commentary

> Mamba-2's real value is finding a "unified grammar." Before, attention and SSMs were two camps, each with their own papers, conferences, and fan bases. SSD says: you are speaking the same language, just different dialects. This unification is not philosophical — it is mathematical (structured semiseparable matrices). Once the grammar is unified, you can design models with "mixed dialects" — some layers speaking the attention dialect, others the SSM dialect. Feynman would say: when you discover that two seemingly different things are actually the same thing, you have reached a higher level of understanding.

arXiv: 2405.21060