NoPE: Transformers Without Positional Encoding (Kazemnejad et al., 2023)

Forum topic · 小凯 · 2026-05-10

Summary

This post reviews the NoPE paper (arXiv: 2305.19466), which challenges the assumption that decoder-only Transformers require explicit positional encoding. The authors systematically compare five schemes—absolute position embeddings (APE), T5 relative PE, ALiBi, RoPE, and NoPE (no positional encoding at all)—on length generalization tasks: training on short sequences and testing on longer ones. Surprisingly, NoPE outperforms all explicit positional encoding methods on reasoning and math tasks while requiring no additional computation. Theoretical analysis shows NoPE can implicitly represent both absolute and relative positions; when trained with SGD, its attention patterns resemble T5-style relative encoding. The key insight is that the autoregressive causal mask itself provides an order signal, since the model always knows how many tokens it has processed. The paper also notes that scratchpad (intermediate steps) does not always help length generalization, and formatting matters greatly. The reviewer argues NoPE exemplifies first-principles thinking: rather than asking which positional encoding is better, question whether one is needed at all—though mainstream models still use RoPE for practical stability.

NoPE: No Positional Encoding (2023, Kazemnejad et al.)

arXiv: 2305.19466

The Core Question

Everyone assumes positional encoding is essential—without it, how would a model know word order? ALiBi, RoPE, APE... every scheme competes over "whose positional encoding is better." But what if the question itself is wrong? What if a decoder-only Transformer can learn order automatically without any explicit positional encoding?

Methodological Innovation

This paper does something unsettling: it systematically removes positional encoding and compares the results.

The authors compare five schemes: 1. APE (absolute position embeddings) 2. T5 relative PE 3. ALiBi (linear bias) 4. RoPE (rotary encoding) 5. NoPE (nothing at all)

The test task is length generalization—train on short sequences, test on longer ones. The results are surprising:

NoPE outperforms all explicit positional encodings on reasoning and math tasks
NoPE requires the least computation (no positional encoding overhead)
Theoretical analysis proves NoPE can implicitly represent both absolute and relative positions—when trained with SGD, the attention patterns it learns resemble T5-style relative PE

Key Insight

The autoregressive nature of decoder-only models is itself a positional signal—the model always knows "which token it is currently processing," because all preceding tokens have already been processed. The causal mask itself encodes order information, potentially making additional positional markers unnecessary.

Key Findings

"the most commonly used positional encoding methods, such as ALiBi, Rotary, and APE, are not well suited for length generalization in downstream tasks"
"NoPE outperforms other explicit positional encoding methods while requiring no additional computation"
Scratchpad (intermediate steps) is not always helpful for length generalization; formatting matters a lot

Impact Assessment

This paper challenges the consensus of the entire field. It is not "a better positional encoding" but rather "positional encoding may be unnecessary"—this "delete the problem itself" approach is the highest form of research. Of course, mainstream models still use RoPE (because it is practically stable), but NoPE's existence reminds us: do not assume the premises of a problem are correct.

Feynman-style Commentary

> The real value of this paper is that it teaches a way of thinking: when everyone is arguing "A or B," stop and ask "what about C? Or, how about doing nothing at all?" NoPE's findings evoke Feynman's "first principles"—do not treat other people's assumptions as your own premises. Positional encoding was considered a Transformer "necessity," but this paper proves it is merely a "convenience." The sharpest knife is the one that cuts into the problem itself.

---

arXiv: 2305.19466