Removing Positional Encoding Helps Transformers Generalize Across Distances: A Counterintuitive Finding
A Surprising Experiment
Imagine training a Transformer on a trivially simple task: copying a token from one position to another with a delay in between. For example, the model sees token "A" at position 5 and must write it out at position 25. During training, it sees delays of 15–25. At test time, it must handle delays of 35, 45, 55—distances it has never seen.
This is the core experiment in the paper *Distance generalization in transformers: why bother with positional encoding?* by Nevermann and Gros (September 2026). They asked three questions:
1. Do positional encoding schemes like RoPE and ALiBi, compared with giving no positional information at all (NoPE), actually help models generalize to new distances? 2. Does more diversity in training distances improve generalization? 3. Does transfer learning (jointly training an auxiliary task) help or hurt distance generalization?
The answer is unexpected: turning off positional encoding makes the model learn better.
What Is "Distance Generalization"?
Two concepts should be distinguished. Length generalization asks: trained on short sequences, can a model handle longer ones? E.g., trained up to 512 tokens, tested at 1024. This is a well-studied problem.
Distance generalization is different: the context length stays fixed, but the "dependency distance" between tokens changes. During training, the model sees "source at position 5, target at position 20" (distance 15); at test time it gets "source at position 5, target at position 40" (distance 35).
This distinction matters. Length generalization fails because the model "can't process such a long context"; distance generalization fails because the model "can read the context but hasn't learned to 'remember across such a gap'." The latter is closer to real scenarios: dialogue systems must recall entities mentioned 20 turns earlier, and code models must understand dependencies spanning hundreds of lines between definitions and call sites.
Experimental Design: Delayed Copy Tasks
The researchers built two synthetic tasks:
- C task (delay copy): copy all 10 source tokens to delayed positions
- S task (selective copy): copy only the even-indexed source tokens
- RoPE (rotary position embedding): the standard choice in current mainstream LLMs (LLaMA, Qwen, etc.)
- ALiBi (attention with linear biases): injects positional information via attention bias
- NoPE: no positional encoding at all, relying only on implicit positional signals from causal attention
During training, the delay distances were 15–25. At test time, distances ranged from 5 to 55. The context length stayed constant—only the gap between source and target changed.
They compared three positional encoding schemes:
Core Finding: NoPE Wins
The result in one sentence: NoPE generalizes best in most cases.
This seems counterintuitive. The entire reason positional encodings exist is to tell the model "where each token is"—why does removing them help?
The researchers' explanation: explicit positional encodings like RoPE and ALiBi hard-code distance information into the attention computation. What the model learns during training is a way of handling "distances 15–25," tied to those distances. When test distances fall outside the training range, the positional encoding's signal becomes an unseen pattern—and an interference.
NoPE is different. Without explicit positional encoding, the model must rely on the implicit mechanisms of causal attention—it has to infer "how far apart" from the data itself. Once this inferential ability is learned, it is more flexible than a hard-coded distance mapping and transfers to new distances.
The authors describe the result as "paradoxical." They also cite similar prior findings on length generalization, where NoPE also performed well. Two independent lines of evidence point to the same conclusion.
The Data-Diversity Paradox
The second finding is equally interesting. The researchers varied the number of training distances—from a single distance, to 5, 10, up to a continuous range.
Result: in absolute terms, more diversity is better; in relative terms, more diversity is worse.
What does this mean? Absolute refers to test accuracy within the training distance range—more diversity means broader coverage and better performance. Relative refers to the marginal gain from each additional training distance—as diversity grows, the marginal benefit of each new distance shrinks, even turning negative.
This matches human learning intuition: going from knowing only C major to G major on piano is huge progress; going from 10 keys to 11 is barely noticeable. Diminishing marginal returns are universal.
There is a subtler finding too: when training distances are too concentrated, the model fails to generalize beyond the training range at all. It learns "how to handle distances 15–25" but not the meta-lesson that "distance itself can vary." With too little diversity, the model treats training distances as constants of the universe.
Transfer Learning: A Double-Edged Sword
The third experiment concerns transfer learning. Models were trained jointly on a main task (e.g., C) and an auxiliary task (e.g., S) to see whether the auxiliary task helps the main task's distance generalization.
Result: sometimes it helps, sometimes it hurts.
When the auxiliary task's distance range is close to the main task's, transfer learning has a positive effect—the "distance-handling" ability learned from the auxiliary task applies directly. But when the auxiliary task's distance range differs too much, transfer becomes harmful—the model is misled into learning inapplicable distance mappings.
The researchers specifically note that moderate out-of-distribution testing mostly benefits from transfer learning, while extreme out-of-distribution testing suffers. This suggests a "sweet spot": the auxiliary and main tasks' distance distributions should overlap, but not coincide completely.
What Does This Mean?
The findings have direct implications for Transformer architecture design:
The role of positional encoding needs re-examination. Mainstream LLMs almost all use RoPE, but this work suggests RoPE may be a bottleneck for distance generalization. If a model must handle token dependency distances unseen during training—extremely common in practice—RoPE's hard-coded distance bias becomes a limitation.
NoPE is not "no positional encoding"—it is "a different positional encoding." NoPE relies on implicit positional signals from causal attention, which is itself an encoding scheme—except what it learns is not "how to handle distance 15" but "how to infer distance from data." The latter is more flexible and transferable.
Data diversity is not a panacea. Increasing coverage of training distances helps, but marginal returns fade fast. What matters more is teaching the model that "distance is variable," rather than exposing it to every distance—neither realistic nor necessary.
Transfer learning has a sweet spot. When choosing auxiliary tasks, the overlap of distance distributions is the key variable. Too close adds nothing; too far causes harm.
Limitations
The study uses synthetic tasks (copying, selective copying), which differ from real language tasks. "Distance dependencies" in natural language are far more complex—semantic, syntactic, and pragmatic dependencies intertwined. Whether NoPE's advantage on synthetic tasks transfers to real language tasks remains to be verified.
The authors also acknowledge their experiments are small-scale (small Transformers); behavior at larger scales may differ. But prior NoPE findings on length generalization have been validated in larger models, so the distance-generalization advantage has a reasonable chance of scaling too.
My Take
The most interesting aspect of this paper is not the "NoPE is better" conclusion itself, but the tension it reveals between architectural bias and generalization. Positional encoding is an inductive bias—you tell the model "position matters, handle it specially." Such biases work within the training distribution but become shackles outside it.
This echoes a classic paradox in deep learning: strong inductive biases help with little data but hurt with lots of it. Convolution's translation invariance, recurrence's temporal assumptions, attention's permutation invariance—all instances of the same problem. NoPE's victory is fundamentally the victory of "letting the data speak": when you stop imposing distance assumptions, the model learns a more flexible treatment of distance from the data itself.
Positional encodings won't disappear—their advantages in training efficiency and inference stability are too obvious. But this research reminds us: components we consider indispensable may just be artifacts of training convenience. Next time a model underperforms on long-context tasks, perhaps the question isn't "how to improve the positional encoding," but "is the positional encoding itself the problem."
---
Paper: Distance generalization in transformers: why bother with positional encoding?
Authors: Daniel Henrik Nevermann, Claudius Gros (Goethe University Frankfurt)
Published: 2026-09-10