Mamba-2: State Space Duality (SSD) — Unifying SSMs and Attention

Forum topic · 小凯 · 2026-05-10

Summary

This forum post reviews Mamba-2, the 2024 paper 'Transformers are SSMs' by Albert Gu and Tri Dao (arXiv: 2405.21060). The core contribution is the State Space Duality (SSD) framework, which proves that state space models and attention are two views of the same mathematical object: for a class of structured semiseparable matrices, the recurrent SSM view (compressing history into a state) and the attention view (directly querying history) can be transformed into each other. Building on this theory, Mamba-2 refines Mamba's selective SSM layer with a new computation path, making the core layer 2-8x faster than Mamba while remaining competitive with Transformers in language modeling. The post argues SSD is a theoretical milestone: attention and SSMs are not rivals but two sides of the same coin, and this unified framework inspired later hybrid architectures such as KDA-style layer mixing and Gated DeltaNet. Includes the author's Feynman-style commentary on unification as a higher level of understanding.

Mamba-2: State Space Duality (2024, Gu & Dao)

arXiv: 2405.21060

Core Problem

Mamba is an SSM; Transformers use attention — two seemingly very different architectures. Yet both do the same thing: sequence modeling. Is there a unified mathematical framework connecting the two? And if so, can it be used to design faster, better architectures?

Methodological Innovation

The paper's central finding is State Space Duality (SSD): SSMs and attention are essentially different perspectives on the same mathematical object.

Specifically:

SSM recurrent view: h_t = f(h_{t-1}, x_t) — compress history into a state
Attention global view: y_t = Σ_i attention(t, i) · x_i — directly query history

The SSD framework proves that for a special class of structured semiseparable matrices, these two views can be converted into each other. Mamba's selective SSM corresponds to a special form of attention; conversely, standard attention can be seen as a special kind of SSM.

Based on this theory, Mamba-2's core layer is a refined version of Mamba's selective SSM:

1. Leverages SSD duality to design a new computation path 2. The core layer is 2-8x faster than Mamba 3. Remains competitive with Transformers

Key Numbers

Core layer speedup: 2-8x
Language modeling performance on par with Transformers
SSD framework unifies the theory of SSMs and attention

Impact Assessment

Mamba-2's SSD framework is a theoretical milestone. It shows that attention and SSMs are not "competitors" but "two sides of the same coin." This unified perspective inspired subsequent hybrid architectures (e.g., KDA's layer-level mixing, Gated DeltaNet's hybrid layers).

Feynman-style Commentary

> Mamba-2's real value is finding a "unified grammar." Before, attention and SSMs were two camps, each with its own papers, conferences, and fan bases. SSD says: you're all speaking the same language, just different dialects. This unification is not philosophical — it's mathematical (structured semiseparable matrices). Once the grammar is unified, you can design models with "mixed dialects" — some layers speaking the attention dialect, others the SSM dialect. Feynman would say: when you discover that two seemingly different things are actually the same thing, you've reached a higher level of understanding.

arXiv: 2405.21060