ZetaGPT: Replacing External Positional Encoding with Recurrent State-Space Dynamics

Forum topic · ✨步子哥 · 2026-08-11

Summary

ZetaGPT is an open-source reference implementation of a positional-encoding-free language model, introducing a causal state-space module (SSM) before self-attention in each Transformer block so that positional information emerges from recurrent hidden-state dynamics rather than being added externally. Architecturally, each block combines a Mamba-style causal SSM, gated multi-head attention to mitigate attention sink, and a standard FFN. The authors train sub-billion configurations (34.4M to 479.9M parameters) end-to-end, including tokenizer training, WikiText-103 pretraining, SFT, reward modeling, RLHF, and GRPO-based chain-of-thought reasoning on GSM8K without supervised CoT data. A key empirical finding is the spontaneous emergence of multi-scale memory horizons across SSM channels within the first 19.7% of training. The paper positions ZetaGPT as a reproducible research baseline for studying whether explicit positional encodings like RoPE are truly necessary, though it acknowledges scale and data limitations and the absence of a direct head-to-head comparison with RoPE models.

Overview

ZetaGPT is an open-source reference implementation of a positional-encoding-free state-space-attention language model authored by Róisín Luo (University of Galway, Ireland). It challenges the assumption that Transformer architectures require explicit positional encodings such as RoPE by letting positional information emerge from recurrent state dynamics inside the model itself.

Paper: https://arxiv.org/abs/2608.09432
Code: https://github.com/roisincrtai/zetagpt

Why positional encoding exists

Self-attention is permutation-equivariant: shuffling the input sequence shuffles attention outputs but leaves values unchanged. Standard solutions (RoPE, ALiBi, etc.) inject position information by modifying query/key computations. This introduces engineering pain — context lengths beyond training require scaling tricks (NTK-aware, YaRN, LongRoPE).

ZetaGPT asks: can positional information grow from the architecture instead of being bolted on?

Core architecture: SSM before attention

Each Transformer block stacks three residual sublayers:

1. Causal state-space module (SSM) — A Mamba-style selective SSM updates a hidden state

$$h_t = a_t \odot h_{t-1} + b_t \odot x_t$$

with input-dependent decay $a_t$ and input gate $b_t$. Token representations exiting this layer are already position-aware. 2. Gated multi-head attention — Standard self-attention over position-aware representations. The gating mechanism mitigates attention sink. 3. Feed-forward network (FFN) — Standard MLP.

The key design choice is SSM before attention: position information is encoded by recurrent state dynamics, not by external rotation matrices.

Emergent multi-scale memory horizons

The authors define a memory horizon $\tau = -(\ln a_t)^{-1}$ and observe that different SSM channels spontaneously specialize:

Short-$\tau$ channels: rapid forgetting, focus on local context.
Long-$\tau$ channels: slow forgetting, maintain long-range dependencies.

This differentiation emerges within the first ~12,800 training steps (19.7% of the budget), without being explicitly specified — analogous to biological neurons with diverse time constants.

Model configurations

| Config | Layers | Dim | Params | |--------|--------|-----|--------| | ZetaGPT-S | 6 | 384 | 34.4M | | ZetaGPT-M | 12 | 768 | 159.2M | | ZetaGPT-L | 24 | 1024 | 479.9M |

All are sub-billion — positioned for research, prototyping, and education rather than frontier competition.

End-to-end training pipeline

1. Tokenizer: Byte-level BPE, vocab size 50,259 (no OOV). 2. Pretraining: WikiText-103, 64,840 steps, lr $2 \times 10^{-5}$; loss 10.90 → 6.05 nats/token. 3. SFT: Alpaca-GPT4, 2,342 steps. 4. Reward model: Alpaca-GPT4, 2,500 steps. 5. RLHF: standard pipeline. 6. Chain-of-thought via GRPO: trained on GSM8K using Group Relative Policy Optimization — reasoning emerges from pure RL, with no supervised CoT data.

The full pipeline runs on a single architecture without switching backbones or adding auxiliary modules.

Why "no positional encoding" matters

Length extrapolation: Recurrence has no hard length cap — position information persists as long as state does not decay.
Architectural cleanliness: Position becomes an internal property rather than an external patch.
Research baseline: With nearly every modern LLM using RoPE, the field lacked a reproducible counter-example.

Gated attention: a side contribution

Gated multi-head attention (Qiu et al. 2025) dynamically modulates attention output, shown in prior work to relieve attention sink and activation collapse on pre-layer-norm Transformers — phenomena linked to hallucination.

Honest limitations

Smallest meaningful scale: max 479.9M params.
Narrow data: WikiText-103 only; no MMLU / HumanEval reporting.
No direct RoPE vs. no-RoPE ablation at matched scale and data.
Single-author, small-team scope; reproducibility requires community effort.

Takeaway

ZetaGPT does not move frontier benchmarks. It demonstrates substitutability of a design choice: positional encoding is not sacred. For researchers exploring architectural alternatives, it offers a clean, fully open baseline across tokenizer, pretraining, SFT, RLHF, and CoT training.