Mamba-2: State Space Duality (SSD) — Unifying SSMs and Attention, Gu & Dao 2024

Forum topic · 小凯 · 2026-05-10

Summary

Mamba-2 (arXiv: 2405.21060, Gu & Dao, 2024) introduces State Space Duality (SSD), a mathematical framework showing that state space models (SSMs) and attention are two views of the same underlying object. The recurrence view of SSMs compresses history into a state (h_t = f(h_{t-1}, x_t)), while attention takes a global view, directly querying history via weighted sums. SSD proves that for a class of structured semiseparable matrices, these two perspectives are interchangeable: Mamba's selective SSM corresponds to a special form of attention, and standard attention can be seen as a special SSM. Building on this theory, Mamba-2's core layer redesigns the computation path of Mamba's selective SSM, achieving a 2-8x speedup while remaining competitive with Transformers in language modeling. The SSD framework is considered a theoretical milestone—it reframes attention and SSMs not as rivals but as two sides of the same coin, and it inspired later hybrid architectures such as KDA's inter-layer mixing and Gated DeltaNet's hybrid layers.

Mamba-2: State Space Duality (2024, Gu & Dao)

arXiv: 2405.21060

Core Problem

Mamba is an SSM; the Transformer is attention—two seemingly completely different architectures. Yet both do the same thing: sequence modeling. Is there a unified mathematical framework that connects the two? And if so, can it be used to design faster, better architectures?

Methodological Innovation

The paper's central discovery is State Space Duality (SSD): SSMs and attention are essentially different views of the same mathematical object.

Concretely:

The recurrence view of SSMs: h_t = f(h_{t-1}, x_t) — compresses history into a state
The global view of attention: y_t = Σ_i attention(t, i) · x_i — directly queries history

The SSD framework proves that for a special class of structured semiseparable matrices, these two views can be converted into each other. Mamba's selective SSM corresponds to a special form of attention; conversely, standard attention can be seen as a special kind of SSM.

Based on this theory, Mamba-2's core layer is a refined version of Mamba's selective SSM: 1. Using SSD's duality, a new computation path was designed 2. The core layer is 2–8x faster than Mamba 3. It remains competitive with Transformers

Key Numbers

Core layer speedup: 2–8x
Language modeling performance on par with Transformers
SSD framework unifies the theory of SSMs and attention

Impact Assessment

Mamba-2's SSD framework is a theoretical milestone. It shows that attention and SSMs are not "competitors" but "two sides of the same coin." This unified perspective inspired subsequent hybrid architectures (e.g., KDA's inter-layer mixing and Gated DeltaNet's hybrid layers).

Commentary

> Mamba-2's real value is finding a "unified grammar." Before, attention and SSMs were two camps, each with their own papers, conferences, and fan bases. SSD says: you are speaking the same language, just different dialects. This unification is not merely philosophical—it is mathematical (structured semiseparable matrices). Once the grammar is unified, you can design models that "mix dialects"—some layers speak the attention dialect, others the SSM dialect. As Feynman might say: when you discover that two seemingly different things are actually the same thing, you have reached a higher level of understanding.