Sessa: Selective State Space Attention — Power-Law Memory by Injecting Attention into Feedback Loops

Forum topic · 小凯 · 2026-04-21

Summary

A Chinese tech forum post analyzes Sessa (Selective State Space Attention), a 2026 sequence-model architecture that injects attention into the feedback loop of state space models. The author explains why Transformer attention dilutes long-range information at O(1/ℓ) while Mamba-style SSMs forget exponentially at O(e^{-cℓ}), and shows that Sessa's multi-hop feedback routing achieves a power-law memory tail of ℓ^{-β}. The post covers the architecture (forward attention, feedback attention with a bounded tanh gain γ_t, and a causal lower-triangular solve (I − B_fb)s = f), the Neumann-series interpretation as aggregation over multi-hop paths, theoretical results on BIBO stability, implicit positional encoding, universal approximation, and flexible selective retrieval profiles requiring depth k ≥ 2. Experiments report Sessa beating Transformer and Mamba2 on SymbolSoup (86.01%), Diffuse MQAR (15.41%), with a modest short-context perplexity trade-off on SimpleStories (8.37 vs 7.67). The dense implementation remains quadratic in sequence length, a noted limitation.

This post from zhichai.net is an in-depth walkthrough of Sessa (Selective State Space Attention) (arXiv:2604.18580v1), a sequence-model architecture that unifies Transformer-style attention with state-space-model (SSM) feedback.

Key points

Two paradigms, two failure modes. Transformers use single-hop direct reads (attention) — powerful but quadratic in cost, and in diffuse-attention regimes each historical token's influence decays as O(1/ℓ) (dilution). Mamba-style SSMs carry a fixed hidden state with linear cost, but unless they can "freeze" their step size Δ_t ≈ 0 ("failed freeze time"), information decays exponentially, O(e^{-cℓ}) (forgetting).
Sessa's core idea. Inject input-dependent attention into the feedback path of a gated MLP block. The Mixer combines:

Forward causal attention

Feedback attention

causal lower-triangular solve

Why power-law memory. Since B_fb is strictly lower-triangular (nilpotent), (I − B_fb)^{-1} = Σ_k B_fb^k aggregates all multi-hop paths from source τ to target t. Under diffuse feedback routing, Theorem 8 gives a tail bound |y_{τ+ℓ}| ≤ C·ℓ^{-β_tail} with β_tail = 1 − γ_max·c₂ ∈ (0,1]; under uniform routing the ℓ^{-β} rate is shown to be tight via Gamma-function asymptotics.
Flexible selective retrieval. Theorem 12 shows Sessa can realize retrieval profiles with margin M ≥ c_−(1+ℓ)^{ν_k(β)}, ν_k(β) = k(1−β) − 1: decaying for k=1, frozen (ν=0) for k ≥ 2 with β = 1−1/k, and even growing (ν>0) for β < 1−1/k. Matching impossibility results (Proposition 13) show depth-limited Transformers and failed-freeze-time Mambas can only decay (≈ (log ℓ)^{N−1}/ℓ and (1+ℓ)^{N−1}e^{-cℓ} respectively).
Additional theory. BIBO stability of the triangular solve when |γ_t| < 1 (Lemma 4.2); the feedback branch implicitly generates absolute positional information (Lemmas 4.10, Corollary 4.13); and universal approximation of continuous causal sequence maps (Theorem 14).

Experimental results

| Task | Sessa | Transformer | Mamba2 | |------|-------|-------------|--------| | SymbolSoup (acc.) | 86.01% | 79.21% | 5.00% (≈ chance) | | Diffuse MQAR (token acc.) | 15.41% | 12.22% | 0.21% | | SimpleStories (PPL) | 8.37 | 7.67 | 7.72 |

Mamba2's failure on long-range tasks matches the "failed freeze time" theory. On short-context language modeling Sessa is slightly worse; an ablation removing the feedback branch improves perplexity to 8.09, confirming the trade-off is capacity allocated to long-range feedback.

Limitations noted

The dense full-prefix implementation remains quadratic in sequence length (mitigated by triangular-solve kernels like TRSM, but not yet validated at scale).
No architecture dominates at all context lengths; adaptive use of feedback may be needed.
Experiments are small-scale synthetic/benchmark tasks; large-corpus and other-modality validation remains future work.

References cited

1. Vaswani et al. (2017), *Attention Is All You Need*, NeurIPS. 2. Gu & Dao (2024), *Mamba*, COLM. 3. Dao & Gu (2024), *Transformers are SSMs*, ICML. 4. Huang et al. (2025), *Understanding input selectivity in Mamba*, ICML. 5. Horbatko (2026), *Sessa: Selective State Space Attention*, arXiv:2604.18580v1.

The post's central takeaway: long-range behavior depends not just on how routing coefficients are generated, but on how they compose over time — single-hop dilution, single-chain exponential decay, and multi-hop power-law tails are qualitatively different regimes, and Sessa's multi-path feedback is what enables the latter.