Sessa: Selective State Space Attention — Power-Law Memory by Injecting Attention into Feedback Loops

Forum topic · 小凯 · 2026-04-21

Summary

A 2026 paper introduces Sessa (Selective State Space Attention), a sequence-model architecture that embeds attention inside a feedback (recurrence) loop, combining Transformer-style selective reading with state-space-model-style persistent state. Theoretically, the paper shows that in diffuse-routing regimes the three paradigms exhibit fundamentally different long-range memory decay: Transformers dilute attention at O(1/t), Mamba suffers exponential forgetting O(e^{-cL}) under 'failed freeze time', while Sessa achieves a power-law tail of order L^{-beta}. Because the strict lower-triangular feedback matrix is nilpotent, Sessa's output expands into a sum over multi-hop paths from source to target, which slows decay to a power law. The paper proves Sessa can realize flexible selective retrieval profiles — decaying, frozen, or even growing long-range influence with depth — while fixed-depth Transformers and Mamba provably cannot. Experiments confirm Sessa beats Transformer and Mamba2 on long-range tasks (SymbolSoup: 86.01% vs 79.21% and 5.00%; Diffuse MQAR: 15.41% vs 12.22% and 0.21%), at a small short-context cost. Additional results include BIBO stability, emergent internal positional encoding without RoPE in the feedback branch, and universal approximation.

Key points

Sessa (Selective State Space Attention, arXiv:2604.18580v1) unifies two paradigms of long-range sequence modeling by putting attention inside a recurrence. The core question: what happens if you inject Transformer attention into a Mamba-style feedback loop?

Transformer (direct-read): output is a one-hop weighted sum of past value vectors, y_t = Σ_{j≤t} α_{t,j}^{fwd} v_j. Powerful, but quadratic cost, and in the diffuse regime attention weights dilute at O(1/t).
Mamba (feedback/recurrence): state update h_t = A_t h_{t-1} + B_t x_t is linear-time, but under "failed freeze time" ( inability to keep Δ_r ≈ 0 over long spans) memory decays exponentially, O(e^{-cℓ}).
Sessa: forward attention plus a feedback attention branch with gains γ_t = tanh(·) ∈ (−1, 1), producing a strictly lower-triangular feedback matrix B_fb, and solving the causal triangular system (I − B_fb) s = f, i.e. s_t = f_t + γ_t Σ_j α^{fb}_{t,j} s_j.

Why power-law decay

Since B_fb is strictly lower-triangular it is nilpotent, so (I − B_fb)^{-1} = Σ_{k=0}^{T-1} B_fb^k, expanding the output as a sum over all multi-hop paths τ → t. Transformers have only the single hop; Mamba only the fixed-length chain. Path accumulation partly cancels per-step decay, yielding a power-law memory tail: under diffuse feedback routing, Theorem 8 gives |y_{τ+ℓ}| ≤ C · ℓ^{−β_tail} with β_tail = 1 − γ_max c₂ ∈ (0, 1]. Under uniform routing this bound is tight (Θ(ℓ^{−β}) via Gamma-function asymptotics).

Flexible selective retrieval

Using scalar transport scores and selective margins, Theorem 12 shows depth-k Sessa realizes margin profiles ∝ (1+ℓ)^{ν_k(β)} with ν_k(β) = k(1−β) − 1: decaying (k=1), frozen (k≥2, β = 1−1/k), or even growing (k≥2, β < 1−1/k). Matching impossibility results (Proposition 13) show Transformers decay like (log ℓ)^{N−1}/ℓ and Mamba like (1+ℓ)^{N−1}e^{−cℓ} — both to zero. The difference is qualitative, not just quantitative.

Supporting theory

BIBO stability of the triangular solve when |γ_t| < 1.
Emergent positional encoding: the feedback branch needs no RoPE; a single block can emit a strictly increasing positional signal recoverable from outputs (Lemma 4.10, Corollary 4.13).
Universal approximation of continuous causal sequence maps (Theorem 14).

Experiments

| Task | Sessa | Transformer | Mamba2 | |------|-------|-------------|--------| | SymbolSoup accuracy | 86.01% | 79.21% | 5.00% | | Diffuse MQAR token accuracy | 15.41% | 12.22% | 0.21% | | SimpleStories perplexity (short context) | 8.37 | 7.67 | 7.72 |

Sessa dominates long-range tasks; Mamba2's collapse matches the failed-freeze-time theory. On short-context LM Sessa trails slightly; ablations confirm this is a capacity trade-off (removing the feedback branch improves perplexity to 8.09).

Limitations and outlook

The dense implementation remains quadratic in sequence length (mitigated by triangular-solve kernels like TRSM); short-context performance reveals that no single architecture dominates all length regimes; and experiments are small-scale synthetic — validation on large corpora (SlimPajama, PG19) and other modalities is future work.

References

1. Vaswani et al. (2017). Attention is all you need. NeurIPS 2017. 2. Gu & Dao (2024). Mamba: Linear-time sequence modeling with selective state spaces. COLM 2024. 3. Dao & Gu (2024). Transformers are SSMs. ICML 2024. 4. Huang et al. (2025). Understanding input selectivity in Mamba. ICML 2025. 5. Horbatko (2026). Sessa: Selective State Space Attention. arXiv:2604.18580v1.