Why Attention-Free Models Still Have "Attention Sinks": A Mathematical Proof of First-Token Attraction in Mamba
> Paper: *First-Token Attraction in Mamba Dynamics* > Authors: T. T. Nguyen et al. (6 authors), arXiv:2611.08303, Nov 2026, cs.AI
1. A Puzzle
Anyone who has worked with Transformers has heard of the strange phenomenon: no matter what you ask, the model dumps a disproportionate share of its attention on the first token—even when that token is just a period or a placeholder. Researchers gave it a vivid name: the attention sink.
The mainstream explanation has an accounting flavor: softmax forces all attention weights to sum to 1, so surplus weight needs somewhere to be dumped, and the first token—"nobody uses it"—becomes the natural landfill. Under this explanation, the sink is a byproduct of softmax—a consequence of the architecture's accounting rules.
Mamba's arrival provided a natural control group. Mamba is a state space model (SSM): no softmax, no weights that must be normalized, and the interaction coefficients between tokens are general matrices—not guaranteed positive, not guaranteed symmetric, no guarantees at all. By the "accounting" theory, there should be no sink here.
Experiments say: there is. And this paper proves it: not a bug, not a coincidence, but a theorem of the dynamical system.
2. The First Token's "Privilege"
To understand why the sink must appear, start with a simple consequence of causality: the first token is the only token in the sequence that "cannot hear anyone else."
Its state update involves only itself (cleanly written in the paper as dx₁/dt = A₁₁·P(x₁) = 0)—so x₁ either stays put or slides toward a small set of discrete attractors (in experiments, mostly "one-hot" directions: unit vectors with a single nonzero component). Meanwhile, the dynamics of every subsequent token contains a reference to x₁.
In other words: the first token becomes the reference frame of the whole system. Everything moves, but every motion is described relative to it. An unmoving reference point has a sharper name in dynamical systems—an attractor.
3. Positivity Is Gone—So What Happens to the Theorem?
The obstacle to a proof is real: existing tools for analyzing attention dynamics (hemisphere Lyapunov methods, Shub-type genericity) all rely on "interaction coefficients being positive scalars"—Mamba's matrix coefficients block that path.
The authors' alternative is the paper's most elegant mathematical move: rather than proving the coefficients are positive, they prove that the dynamics itself grows an equivalent positivity. They study the high-dimensional "invariant box" formed by the interaction variables ⟨xᵢ(t), S_BC x_j(t)⟩—trajectories starting inside the positive box can never leave it (by contradiction using uniqueness of ODE solutions); a "sharp exit-time argument" then recovers effective sign determinism. The guarantee the architecture lacks is supplied by the geometry of the equations.
The convergence proof then unfolds like a layered campaign: first reduce the L-token system to pairwise couplings, seal off escape routes with the invariant box, advance layer by layer by induction, and finally handle the subtlest step—showing that the "orthogonal drift" equilibria are all saddles with measure-zero basins of attraction. The conclusion (for almost every initialization):
> All tokens converge to ±x₁(0) at an exponential rate: ∑ᵢ|xᵢ(t) − sᵢx₁(0)| ≲ e^(−C₀t), sᵢ ∈ {±1}.
Note the ±1—it precisely predicts the experimentally observed "one-hotting": tokens don't cluster randomly, but line up along the reference direction or its opposite.
4. Prediction and Verification
The test on Falcon-Mamba-7B is clean-cut: measuring the hidden "attention" matrices layer by layer, early layers show roughly uniform token distributions, but with depth, the concentration of almost all tokens on the first token steadily rises, dominating the last few layers. Theory says "attraction accumulates with depth"; the experimental sink curves climb with depth exactly so. In simplified Mamba-2 blocks, most first tokens indeed converge to one-hot directions—the ±x₁ prediction is fulfilled verbatim.
This is the first time the attention sink phenomenon has been both observed and proven in a state space model. It shows the sink is not created by softmax's accounting—it is a product of causal structure itself: as long as every member of a sequence's dynamics references an initial member that "hears no one," clustering happens, whether the coefficients are scalars or matrices, normalized or not.
5. Aftermath: Reference Frames, Attractors, and the Retirement of the "Landfill Theory"
Looking back, the "softmax weights need somewhere to dump" landfill theory explained the phenomenon's existence but not its inevitability. The new proof gives a more fundamental picture: the sink is an attractor, not a garbage bin. The first token attracts everything not because "someone has to take out the trash," but because it is the only one that doesn't move—and an unmoving reference point is naturally a destination.
The implications extend beyond Mamba: the paper's framework applies directly to causal attention, classical time-series models, and other sequence dynamics. For engineers, it leaves a concrete question: since token clustering toward the first token is the universal fate of deep SSMs, shouldn't the empirical "first-token special" patches in KV caching, quantization, and long-context design be re-examined through attractor theory?
The Feynman-esque answer is always: first figure out what it is, then decide what to do about it. This time, "what it is" now has an answer—a theorem with an exponential convergence rate.
---
Further reading
1. Nguyen et al., *First-Token Attraction in Mamba Dynamics*, arXiv:2611.08303 (2026) 2. Gu, Dao, *Mamba: Linear-Time Sequence Modeling with Selective State Spaces* (2023, the Mamba architecture) 3. Xiao et al., *Efficient Streaming Language Models with Attention Sinks* (2023, the sink phenomenon and StreamingLLM on the Transformer side) 4. Falcon-Mamba-7B (TII, 2024, the experimental platform used in this paper)
*This article is part of the "Zhichai · Mathematical Foundations and Frontier Intersections" series, Season 5, Round 5.*