Sessa: A Deep Dive into the Selective State-Space Attention Architecture

Background and Motivation

Modern sequence modeling is dominated by two architectures: Transformers and structured state-space models (SSMs). Transformers can access any element in the context via self-attention, but when attention becomes diffuse (spreading uniformly over many past tokens), the influence of any single token is diluted, typically decaying as O(1/ℓ) with distance. Structured SSMs propagate information through an explicit recurrent state, but even selective SSMs like Mamba suffer exponential decay of long-range sensitivity when they cannot maintain "frozen time."

Both mechanisms thus struggle in long-context settings: Transformers lose distant information through attention dilution, while SSMs may forget long-range dependencies through state decay. Existing architectures either retrieve the past through a single attention read or propagate information through a single feedback chain, lacking the ability to aggregate multiple paths within a single layer.

Core Innovation: Attention in the Feedback Path

Sessa's central innovation is placing attention inside the feedback path. It injects input-dependent attention routing into the recurrent feedback loop, combining direct reads of the input with aggregation through feedback state. This enables multi-path aggregation within a single layer: past tokens can influence future states through multiple attention paths rather than a single read or a single recurrent chain.

Concretely, each Sessa decoder layer uses a single gated-MLP-style module that combines the causal forward attention signal with a feedback term. The feedback term mixes past outputs through a lower-triangular linear system (causal convolution), enabling multi-hop routing. This feedback-solve mechanism is the key to Sessa's long-range memory properties.

Theoretical Analysis: Long-Range Memory and Selective Retrieval

Power-Law Memory Tail

Under stated assumptions and matched conditions, Sessa achieves a power-law memory tail: influence decays as O(ℓ^-β) with 0<β<1. This is much slower than the Transformer's O(1/ℓ) and Mamba's exponential decay when frozen time fails.

The mechanism: Sessa's feedback can be written as a strictly lower-triangular matrix B_fb, whose transport kernel K = (I - B_fb)^-1 is lower triangular with unit diagonal. By constraining feedback routing coefficients appropriately, |K_t,k| for distant targets is bounded by C_K (t-k)^(-β_tail) with 0 < β_tail < 1. In the extreme settings of uniform routing and constant positive feedback, this power-law decay is proven tight: influence decays exactly as Θ(ℓ^-β_tail), so the decay cannot be faster.

Flexible Selective Retrieval

Selective retrieval means the model can dynamically choose which past information to emphasize instead of uniformly attenuating all history. By accumulating information across multiple routing depths and multiple temporal paths, Sessa can maintain non-decaying influence on distant information when needed, while decaying elsewhere. Theoretical analysis indicates only Sessa among the compared model classes achieves both power-law decay and non-decaying configurations: Transformers cannot hold constant influence over old tokens, and Mamba's long-range sensitivity collapses once it exits a frozen-time regime.

Comparison with Transformer and Mamba

• Memory decay rate: Transformer O(1/ℓ); Mamba O(e^-λℓ) when frozen time fails; Sessa O(ℓ^-β), 0<β<1 — the slowest decay at large ℓ.
• Selective retrieval: Transformer influence always dilutes with distance; Mamba retains short-term information via gating but loses long-range sensitivity in non-frozen regimes; Sessa accumulates influence through feedback even with very diffuse attention routing.
• Mechanism: Transformer relies on global self-attention with quadratic cost in sequence length; Mamba processes sequences in linear time but needs extra mechanisms for long-range retention; Sessa combines global attention (direct input reads) with local feedback (past-output aggregation) in a single layer, achieving linear-time complexity with better memory characteristics.

*Figure 1 in the original post compares long-range memory decay: Sessa (β=0.5) follows a power law, Transformer a linear decay, and Mamba an exponential decay.*

Experimental Evaluation

Experiments validate the theoretical claims. On long-context benchmarks, Sessa achieves the strongest performance under matched architecture and training budgets, indicating more effective retention and use of distant information. On short-context language modeling, Sessa remains competitive with Transformer and Mamba-style baselines, showing it does not trade short-term performance for long-range memory.

Architecture Details and Implementation

Sessa is designed as a plug-and-play replacement for standard decoder blocks. Instead of alternating attention and MLP blocks, each layer wraps a recurrent mixer in a single gated-MLP-style block. The mixer merges causal forward attention with a feedback term computed via lower-triangular matrix multiplication (multi-hop routing).

To bound feedback gain and ensure stability, Sessa introduces a hyperparameter γ_max < 1 requiring the feedback gain |γ_t| < γ_max at all times, preventing instability and gradient explosion. The implementation is based on PyTorch with optional FlashAcceleration (FlashAttention) support. Simply replacing existing decoder layers with Sessa layers yields stronger long-range information retention.

Conclusion

By embedding attention in the feedback path, Sessa addresses the long-context limitations of both Transformers and SSMs. It demonstrates the feasibility of multi-path aggregation within a single layer and shows, both theoretically and empirically, slower long-range decay and more flexible selective retrieval. This opens new directions for information retention in long-context NLP, time-series forecasting, and beyond, and suggests a broader family of hybrid feedback-attention models to come.