NoPE: Transformers Without Positional Encoding Can Generalize to Longer Sequences (Kazemnejad et al., 2023)

Forum topic · 小凯 · 2026-05-10

Summary

NoPE (arXiv: 2305.19466, Kazemnejad et al., 2023) challenges the assumption that explicit positional encoding is necessary for decoder-only Transformers. The authors systematically compare five schemes—absolute position embeddings (APE), T5 relative PE, ALiBi, RoPE, and NoPE (no positional encoding at all)—on length generalization: training on short sequences and testing on longer ones. Surprisingly, NoPE outperforms all explicit positional encodings on reasoning and math tasks while requiring no additional computation. The paper provides theoretical analysis showing that NoPE can implicitly represent both absolute and relative positions, and that SGD-trained attention patterns resemble T5's relative encoding. The key insight is that decoder-only autoregression itself provides a positional signal: the causal mask means the model always knows how many tokens have been processed, implicitly encoding order information. The authors also note that scratchpad-style intermediate steps do not always help length generalization, with formatting playing a large role. While mainstream models still use RoPE for practical stability, NoPE demonstrates the value of questioning a field's core assumptions rather than optimizing within them.

NoPE: No Positional Encoding (2023, Kazemnejad et al.)

arXiv: 2305.19466

The Core Question

Everyone assumes positional encoding is essential—without it, how would a model know word order? ALiBi, RoPE, APE... every scheme competes to be "the better positional encoding." But what if the question itself is wrong? What if a decoder-only Transformer can learn sequence order automatically without any explicit positional encoding?

Method

The paper does something unsettling: it systematically removes positional encoding and compares the results.

Five schemes are compared: 1. APE (absolute position embeddings) 2. T5 relative PE 3. ALiBi (linear bias) 4. RoPE (rotary encoding) 5. NoPE (nothing at all)

The test task is length generalization—train on short sequences, test on longer ones. The results are surprising:

NoPE outperforms all explicit positional encodings on reasoning and math tasks
NoPE requires the least computation (no positional encoding overhead)
Theoretical analysis proves NoPE can implicitly represent both absolute and relative positions—when trained with SGD, its learned attention patterns resemble T5's relative PE

Key Insight

The autoregressive nature of decoder-only models is itself a positional signal—the model always knows "which token it is currently processing," because all preceding tokens have already been handled. The causal mask itself encodes order information, potentially making additional positional markers unnecessary.

Key Findings

"the most commonly used positional encoding methods, such as ALiBi, Rotary, and APE, are not well suited for length generalization in downstream tasks"
"NoPE outperforms other explicit positional encoding methods while requiring no additional computation"
Scratchpad (intermediate steps) is not always helpful for length generalization; formatting matters a lot

Impact Assessment

This paper challenges the consensus of an entire field. It is not "a better positional encoding" but "positional encoding may be unnecessary"—this kind of "deleting the question itself" thinking is the highest form of research. Of course, mainstream models still use RoPE today (because it is practically stable), but the existence of NoPE reminds us: do not assume the premises of a problem are correct.

Feynman-Style Commentary

> The real value of this paper is that it teaches you a way of thinking: when everyone is arguing whether "A or B is better," stop and ask—"What about C? Or, what about doing nothing at all?" NoPE's discovery echoes Feynman's "first principles"—do not treat other people's assumptions as your own premises. Positional encoding was considered a Transformer "necessity," but this paper proves it is merely a "convenience." The sharpest blade is the one that cuts open the question itself.