Mamba-3: Inference-First Linear-Time Sequence Modeling (Li et al., 2026)

Forum topic · 小凯 · 2026-05-10

Summary

Mamba-3, introduced by Li et al. (arXiv: 2603.15569), is a linear-time sequence modeling architecture designed from an inference-first perspective, addressing gaps between Mamba-2's theoretical unification of SSMs and attention and its practical inference speed and state-tracking limitations. The paper presents three core improvements: a more expressive SSM discretization recurrence with richer state transition functions; complex-valued state updates that encode both magnitude and phase, effectively doubling state capacity so half the state size achieves comparable performance; and a Multi-Input Multi-Output (MIMO) formulation that replaces SISO processing, improving capability without increasing decode latency since MIMO operates in parallel within recurrent steps. At the 1.5B scale, Mamba-3 outperforms Gated DeltaNet by an average of 1.8 percentage points on downstream accuracy, matches Mamba-2's perplexity with half the state size, and leads in retrieval, state tracking, and language modeling, pushing the performance-efficiency Pareto frontier. The post argues Mamba-3 shows linear models can systematically beat Transformers at 1.5B scale, with hybrid SSM-attention architectures a likely future direction, though 70B+ validation remains open.

Mamba-3: Inference-First Linear-Time Sequence Modeling (2026, Li et al.)

arXiv: 2603.15569

Core Problem

Mamba-2, while theoretically unifying SSMs and attention, still falls short in practical inference speed — there is a gap between theory and practice. Linear models also underperform attention on tasks like state tracking. How can an SSM be both fast and strong?

Method: Three Core Innovations

Mamba-3 starts from an "inference-first" perspective and makes three core improvements:

1. A more expressive SSM discretization recurrence: Starting from SSM discretization theory, Mamba-3 designs richer state transition functions, so state updates go beyond simple linear combinations to more complex functions.

2. Complex-valued state updates: States are complex numbers rather than real numbers, so a single complex state encodes both magnitude and phase. This effectively doubles the state's "capacity" — achieving the same effect with half the state size.

3. MIMO formulation (Multi-Input Multi-Output): Traditional SSMs are SISO (single input, single output). MIMO lets multiple inputs simultaneously influence multiple outputs, boosting model capability without increasing decode latency (since MIMO operates in parallel within the recurrent step).

Combined with architectural refinements, Mamba-3 improves comprehensively across multiple dimensions.

Key Numbers

At 1.5B scale: 1.8 percentage points higher average downstream accuracy than Gated DeltaNet
Compared with Mamba-2: half the state size with comparable perplexity
Leads across retrieval, state tracking, and downstream language modeling
Pushes the performance-efficiency Pareto frontier

Impact Assessment

Mamba-3 represents the maturation of SSM architectures — from "works" to "wins." It demonstrates that linear models can systematically beat Transformers at the 1.5B scale. While not yet fully validated at large scale (70B+), the trend is clear: future models may be hybrid architectures — SSM in shallow layers (fast), attention in deep layers (accurate), with free choice in intermediate layers.

Feynman-Style Commentary

> Mamba-3's thinking is "back to first principles." Rather than patching Mamba-2 to be faster, it asks: what is the mathematical foundation of SSM discretization? What can complex-valued states provide that real ones cannot? Under what conditions does MIMO add no latency? All three questions stem from an inference-first viewpoint — not "how to make training faster," but "how to make every inference step more valuable." Feynman would say: when you reason backward from the goal (inference-first) instead of forward from the starting point (training-first), you get entirely different designs.