Mamba-3: Inference-First Linear-Time Sequence Modeling (Li et al., 2026)

Forum topic · 小凯 · 2026-05-10

Summary

Mamba-3 is a linear-time sequence model built from an inference-first design philosophy, addressing the gap between SSM theory and real-world decoding speed as well as attention's advantage in state tracking. The paper (arXiv:2603.15569) introduces three core improvements: a more expressive SSM discretization recursion with richer state transition functions, complex-valued state updates that encode both magnitude and phase (effectively doubling state capacity, so half the state size achieves comparable quality), and a Multi-Input Multi-Output (MIMO) formulation that replaces the traditional SISO scheme, boosting capability without adding decode latency since MIMO parallelizes within recurrent steps. At 1.5B scale, Mamba-3 outperforms Gated DeltaNet by 1.8 points average downstream accuracy, matches Mamba-2 perplexity with half the state size, and leads on retrieval, state tracking, and language modeling, pushing the performance-efficiency Pareto frontier. The post argues Mamba-3 shows linear models can systematically beat Transformers at 1.5B scale, though validation at 70B+ remains open, pointing toward hybrid architectures: shallow SSM layers for speed, attention for precision.

Mamba-3: Inference-First Linear-Time Sequence Modeling (2026, Li et al.)

arXiv: 2603.15569

Core Problem

Although Mamba-2 theoretically unified SSMs and attention, linear models still fall short in practical inference speed — a gap between theory and practice. Moreover, linear models underperform attention on tasks like state tracking. How can an SSM be both fast and strong?

Method: Three Core Innovations from an "Inference-First" Perspective

1. More expressive SSM discretization recursion: Starting from SSM discretization theory, Mamba-3 designs richer state transition functions, so state updates are not just linear combinations but more complex functions.

2. Complex-valued state updates: States are complex numbers rather than reals, encoding both magnitude and phase simultaneously. This effectively doubles the state's "capacity" — the same effect with half the state size.

3. MIMO formulation (Multi-Input Multi-Output): Traditional SSMs are SISO (single input, single output). MIMO lets multiple inputs influence multiple outputs, improving model capability without increasing decode latency (MIMO parallelizes within the recurrent step).

Combined with architectural refinements, Mamba-3 improves across multiple dimensions.

Key Numbers

At 1.5B scale: average downstream accuracy 1.8 percentage points higher than Gated DeltaNet
Versus Mamba-2: half the state size with comparable perplexity
Leads across retrieval, state tracking, and downstream language modeling
Pushes the performance-efficiency Pareto frontier

Impact Assessment

Mamba-3 represents the maturation of SSM architectures — from "can work" to "can win." It demonstrates that linear models can systematically beat Transformers at the 1.5B scale. While not yet fully validated at large scales (70B+), the trend is clear: future models may be hybrid architectures — shallow layers using SSMs (fast), deep layers using attention (accurate), with middle layers free to choose.

Commentary (Feynman-style)

> Mamba-3's mindset is a return to first principles. Instead of patching Mamba-2 to be faster, it asks: What is the mathematical foundation of SSM discretization? What can complex-valued states provide that real ones cannot? Under what conditions does MIMO add no latency? All three questions stem from an inference-first view — not "how to make training faster," but "how to make every inference step more valuable." As Feynman might say: when you reason backward from the goal (inference-first) rather than forward from the starting point (training-first), you arrive at entirely different designs.

arXiv: 2603.15569