NoPE: Transformers Don't Need Positional Encodings (Kazemnejad et al., 2023)

Forum topic · 小凯 · 2026-05-10

Summary

NoPE (arXiv: 2305.19466, Kazemnejad et al., 2023) challenges the assumption that positional encodings are required in decoder-only Transformers. The paper systematically compares five schemes—absolute positional embeddings (APE), T5 relative PE, ALiBi, RoPE, and NoPE (no positional encoding at all)—on length generalization tasks: training on short sequences and testing on longer ones. Surprisingly, NoPE outperforms all explicit positional encoding methods on reasoning and math tasks while requiring zero additional computation. The authors provide theoretical analysis showing that NoPE can implicitly represent both absolute and relative positions: the autoregressive causal mask itself acts as a positional signal, and attention patterns learned via SGD resemble T5-style relative encodings. The paper also notes that widely used methods like ALiBi, Rotary, and APE are not well suited for length generalization, and that scratchpad intermediate steps are not always helpful, with formatting playing a large role. While mainstream models still use RoPE for practical stability, NoPE demonstrates that questioning a field's shared assumptions—rather than competing on better variants—can be the most valuable research direction.

NoPE: No Positional Encoding (2023, Kazemnejad et al.)

arXiv: 2305.19466

Core question

Everyone assumes positional encoding is necessary—without it, how would a model know word order? ALiBi, RoPE, APE... each scheme competes on "whose positional encoding is better." But what if the question itself is wrong? What if a decoder-only Transformer doesn't need any explicit positional encoding to learn order automatically?

Method

This paper did something unsettling: it systematically removed positional encodings, then compared performance.

The authors compared five schemes: 1. APE (absolute positional embeddings) 2. T5 relative PE 3. ALiBi (linear bias) 4. RoPE (rotary encoding) 5. NoPE (nothing at all)

The test task was length generalization—train on short sequences, test on longer ones. The results were surprising:

NoPE outperformed all explicit positional encodings on reasoning and math tasks
NoPE requires the least computation (no positional-encoding overhead)
Theoretical analysis proves NoPE can implicitly represent both absolute and relative positions—when trained with SGD, the attention patterns it learns resemble T5's relative PE

Key insight

The autoregressive nature of decoder-only models is itself a positional signal—the model always knows "which token it is currently processing," because all preceding tokens have already been processed. The causal mask itself encodes order information, possibly making extra positional markers unnecessary.

Key findings

"The most commonly used positional encoding methods, such as ALiBi, Rotary, and APE, are not well suited for length generalization in downstream tasks"
"NoPE outperforms other explicit positional encoding methods while requiring no additional computation"
Scratchpad (intermediate steps) is not always helpful for length generalization; formatting matters a lot

Impact

This paper challenges the field's consensus. It is not "a better positional encoding" but "positional encodings may be unnecessary"—the kind of "delete the problem itself" thinking is top-tier research. Of course, mainstream models still use RoPE (because it is practically stable), but NoPE's existence reminds us: don't assume the premises of a problem are correct.

Feynman-style commentary

> The real value of this paper is that it teaches a way of thinking: when everyone is arguing "is A better or B better," stop and ask "what about C? Or, what about doing nothing at all?" NoPE's finding echoes Feynman's "first principles"—don't treat other people's assumptions as your own premises. Positional encoding was considered a Transformer "necessity," but this paper proves it is merely a "convenience." The sharpest knife is the one that cuts into the problem itself.

---

arXiv: 2305.19466