NoPE: Models With Halved or Removed Positional Encoding Get Better at Needle-in-a-Haystack

"I like you" and "You like me" look identical to self-attention: permutation equivariance means shuffling word order changes nothing in attention scores. So the common wisdom since the 2017 Transformer has been that positional information must be injected externally. Over the past four years, that assumption has been dismantled—and the latest outcome is now shipping in production models.

Key points

• Deleting positional encoding entirely works (in decoders). In 2022, Haviv et al. trained GPT-3-XL-style models from 125M to 1.3B parameters without positional encoding. On the Pile, perplexity differed from learnable-PE baselines by only 0.05 (13.10 vs 13.05); on WikiText-103 the gap was 0.55. The reason is the causal mask: a token that can see only *t* previous tokens knows it is the *t*-th token by counting its visible range. This is why it only works for decoder-only models—bidirectional encoders without PE truly degenerate into bag-of-words.
• The math (Kazemnejad et al., 2023). Two theorems: (1) there exist weights by which a NoPE model's first layer recovers absolute positions 1..T into hidden states; (2) if that information survives, later attention dot products decompose into a content term plus a relative-distance term—the mathematical form of relative positional encoding. Note the word *exist*: this capacity is not free—it consumes first-layer compute and residual-stream bandwidth, and requires later layers not to overwrite it. Explicit encodings like RoPE inject per layer without those costs.

Industry doesn't use pure NoPE—half of it

Since 2025, Llama 4, Cohere Command A, and HuggingFace SmolLM3 all keep one position-free full-attention layer per four layers; the rest have positional encoding (Llama 4 and Command A use sliding-window + RoPE layers; SmolLM3 uses full attention + RoPE). Confirmed from config files: Llama 4's source defaults to "one NoPE layer every 4 layers"; Command A's model card states three RoPE sliding-window layers handle local context and a fourth, PE-free global layer handles unlimited sequence-wide interaction.

Cohere ran the largest systematic study: analysis at 750B tokens, then pretraining at 5T tokens. They found an emergent division of labor nobody designed: NoPE layers became global retrieval specialists—attention concentrates exactly on the needle in long contexts—while RoPE layers became local aggregation specialists. Building on this, they hard-capped RoPE-layer attention windows at 8192 while NoPE layers kept full attention: long-context scores *improved*. Conversely, stretching RoPE frequency parameters to extend length degraded NoPE-layer attention quality (needle attention share fell from 0.0765 to 0.0369).

Three pitfalls of pure NoPE

1. Runaway generation after post-training. Qwen3.8's ablations: pretraining metrics are nearly identical with or without RoPE, but after post-training, the NoPE version never stops generating. Qwen's economical fix: keep RoPE on only the first 64 of 256 head dimensions—25% dose, enough to learn when to stop. 2. Attention entropy over long contexts. More tokens in the softmax dilutes attention. Fixable by tuning: raising the attention score scale from 1/√d to 1.2/√d (lower temperature) lets a 2K-trained model generalize to 4K; 1.8 gives 8×. The same trick does not help RoPE, which has its own out-of-distribution position problem. 3. Training-curve instability. Pure NoPE from scratch shows consistently worse perplexity throughout training; at 1B+ parameters with parallel architectures and long context, unrecoverable loss spikes appear. Fixes: QK-Norm suppresses spikes but converges to higher loss; or 10%-dimension partial RoPE matches full RoPE's training loss with 10× less RoPE cache overhead.

Together these point to a philosophical shift, articulated by the DroPE paper: positional encoding is not an organ of the model but scaffolding for training—use it to guarantee convergence, then remove it. Dropping PE entirely after training leaves in-distribution perplexity unchanged and improves stability at longer contexts.

The other road: position lives in structure

• Jamba (the first production Transformer + Mamba SSM + MoE hybrid): Mamba layers process tokens recursively and inherently know order. The paper states plainly that with Mamba layers, mechanisms like RoPE are unnecessary—its attention layers carry no positional encoding, with ablations showing similar results with or without RoPE.
• Kimi Linear mixes Kimi Delta Attention (KDA) and full attention 3:1. KDA compresses new information into a fixed-size memory matrix, comparing old/new before writing, with a forget gate controlling decay—upgraded from a scalar forgetting coefficient to a per-channel vector so each channel decays at its own rate. The paper's elegant contrast: RoPE is fixed, orthogonal, index-based rotation with per-dimension angular speeds; KDA's per-channel decay provides the same "different dimensions, different rhythms" positional sense but learnable and content-driven. Both works show recurrent structures and causal linear attention are two writings of the same thing.
• ZetaGPT, a fresh teaching project, places an SSM module before every attention layer so recursive state writes order into token representations. It admits its tests/ablations are thin and has only a dozen-odd stars, but covers the full pipeline—data prep, tokenizer training, pretraining, post-training, diagnostics—good as a hands-on specimen.

Looking back

The original dogma—no injected positions, no word order—has a corrected form four years later: word order can live in the causal mask, in recurrent structure, or in just a quarter of the layers. Positional encoding has been demoted from mandatory parameter to training-time scaffolding plus a local tool—and in Cohere's 5-trillion-token run, global retrieval was done best precisely by the layers without positional encoding.