NoPE: Decoder-Only Transformers May Not Need Positional Encoding (Kazemnejad et al., 2023)

Forum topic · 小凯 · 2026-05-10

Summary

NoPE (arXiv: 2305.19466) challenges the assumption that Transformers require explicit positional encoding. The authors systematically compare five schemes — absolute positional embeddings (APE), T5 relative PE, ALiBi, RoPE, and No Positional Encoding (NoPE) — on length generalization tasks (training on short sequences, testing on longer ones). Surprisingly, NoPE outperforms all explicit positional encoding methods on reasoning and math tasks while adding zero computational overhead. Theoretical analysis shows NoPE can implicitly represent both absolute and relative positions: trained with SGD, its learned attention patterns resemble T5-style relative encoding. The key insight is that decoder-only autoregression itself provides a positional signal — the causal mask ensures the model always knows how many tokens have been processed. The paper also finds that common methods like ALiBi, RoPE, and APE are not well suited to length generalization, and that scratchpad effectiveness depends heavily on format. Though mainstream models still use RoPE for practical stability, NoPE demonstrates that questioning a problem's premises can be more powerful than optimizing within them.

3. NoPE: No Positional Encoding (2023, Kazemnejad et al.)

arXiv: 2305.19466

The Core Question

Everyone assumes positional encoding is necessary — without it, how would a model know word order? ALiBi, RoPE, APE... every scheme competes on "whose positional encoding is better." But what if the premise itself is wrong? What if a decoder-only Transformer can learn order automatically without any explicit positional encoding?

Method

The paper does something unsettling: it systematically removes positional encoding and compares results. The authors compare five schemes:

1. APE (absolute positional embeddings) 2. T5 relative PE 3. ALiBi (linear bias) 4. RoPE (rotary encoding) 5. NoPE (nothing at all)

The test task is length generalization — train on short sequences, test on longer ones. The results are surprising:

NoPE outperforms all explicit positional encodings on reasoning and math tasks
NoPE requires the least computation (no positional encoding overhead)
Theoretical analysis proves NoPE can implicitly represent both absolute and relative positions — when trained with SGD, its learned attention patterns resemble T5-style relative PE

Key Insight

The autoregressive nature of decoder-only models is itself a positional signal — the model always knows "which token am I currently processing," because all preceding tokens have already been processed. The causal mask itself encodes order information, possibly making additional positional markers unnecessary.

Key Findings

"The most commonly used positional encoding methods, such as ALiBi, Rotary, and APE, are not well suited for length generalization in downstream tasks"
"NoPE outperforms other explicit positional encoding methods while requiring no additional computation"
Scratchpad (intermediate steps) is not always helpful for length generalization — format matters a lot

Impact

This paper challenges the field's consensus. It is not "a better positional encoding" but "positional encoding may be unnecessary" — deleting the problem itself is the highest form of research. Of course, mainstream models still use RoPE (because it is practically stable), but NoPE's existence reminds us: do not assume the problem's premises are correct.

Feynman-style Commentary

> The real value of this paper is that it teaches a way of thinking: when everyone is debating "A or B," stop and ask, "What about C? Or... doing nothing at all?" NoPE's finding evokes Feynman's "first principles" — do not take other people's assumptions as your own premises. Positional encoding was considered a Transformer "necessity," but this paper proves it is merely a "convenience." The sharpest knife is the one that cuts open the problem itself.