Removing Positional Encoding Helps Transformers Learn Better: A Counterintuitive Finding on Distance Generalization
An Experiment That Surprised Everyone
Imagine training a Transformer on an extremely simple task: copying a token across a distance. The model sees token "A" at position 5 and must write it out at position 25. During training, it sees all gaps between 15 and 25. At test time, you give it gaps of 35, 45, 55 — distances it has never seen.
This is the core experiment in the paper *Distance generalization in transformers: why bother with positional encoding?* by Nevermann and Gros (published September 2026). They asked three questions:
1. Do positional encoding schemes like RoPE and ALiBi actually help models generalize to new distances compared to providing no positional information at all (NoPE)? 2. Does seeing more diverse training distances improve generalization? 3. Does transfer learning (jointly training an auxiliary task) help or hurt distance generalization?
The answer is surprising: turning off positional encoding makes the model learn better.
What Is "Distance Generalization"?
First, distinguish two concepts. Length generalization asks: if a model is trained on short sequences, can it handle longer ones? E.g., trained up to 512 tokens, tested on 1024. This is a well-studied problem.
Distance generalization is different: the context length stays fixed, but the "dependency distance" between tokens changes. During training the model sees "source token at position 5, target at position 20" (distance 15); at test time it gets "source at position 5, target at position 40" (distance 35).
This distinction matters. Length generalization failure means "can't process that long a context"; distance generalization failure means "can read the context, but never learned to 'still remember something this far away.'" The latter is closer to real scenarios: dialogue systems must recall entities mentioned 20 turns ago, and code models must understand dependencies spanning hundreds of lines between function definitions and calls.
Experimental Design: Delayed Copy Tasks
The researchers built two synthetic tasks:
- C task (delay copy): copy all 10 source tokens to delayed positions
- S task (selective copy): copy only even-indexed source tokens
- RoPE (rotary position embedding): the standard choice in current mainstream LLMs (LLaMA, Qwen, etc.)
- ALiBi (linear bias attention): injects position information via attention biases
- NoPE: no positional encoding at all, relying solely on the implicit positional signal of causal attention
During training, the delay distances were 15-25. At test time, distances ranged from 5 up to 55. The context length stayed constant; only the gap between "source" and "target" changed.
They compared three positional encoding schemes:
Core Finding: NoPE Wins
The result in one sentence: NoPE generalizes best in most cases.
This sounds counterintuitive. The entire reason positional encodings exist is to tell the model "where a token is" — why would giving no positional information work better?
The researchers' explanation: explicit encodings like RoPE and ALiBi hard-code distance information into the attention computation. During training, the model learns "how to handle distances 15-25," and this handling is bound to those distances. When the test distance exceeds the training range, the positional encoding's signal is an unseen pattern — it becomes interference rather than help.
NoPE is different. Without explicit positional encoding, the model can only rely on implicit mechanisms of causal attention — it must infer "how far apart" from the data itself. Once this inference ability is learned, it is more flexible than a hard-coded distance mapping and transfers to new distances.
The researchers describe the result as "paradoxical." They also cite prior similar findings on length generalization, where NoPE also performed well. Two independent lines of evidence point to the same conclusion.
The Data Diversity Paradox
The second finding is equally interesting. The researchers varied the number of training distances — from just 1 distance, to 5, 10, up to a continuous range.
Result: in absolute terms, more diversity is better; in relative terms, more diversity is worse.
How to understand this? Absolute value refers to test accuracy within the training distance range — more diversity means broader coverage and better performance. Relative value refers to the marginal benefit per additional training distance — as diversity grows, the marginal benefit of each new distance shrinks, even turning negative.
This matches human learning intuition: going from knowing only C major to learning G major on piano is huge progress; going from 10 keys to 11 barely registers. Diminishing marginal returns are universal.
But there is a subtler finding: when training distances are too concentrated, the model cannot generalize beyond the training range at all. The model learns "how to handle distances 15-25" but never learns "that distance itself can vary." With too little diversity, the model treats its training distances as a cosmic constant.
Transfer Learning: A Double-Edged Sword
The third experiment examined transfer learning. The model trains jointly on a main task (e.g., C) and an auxiliary task (e.g., S), testing whether the auxiliary task helps the main task's distance generalization.
Result: sometimes it helps, sometimes it hurts.
When the auxiliary task's distance range is close to the main task's, transfer learning has positive effects — the distance-handling skills learned from the auxiliary task apply directly. But when the auxiliary task's distance range differs too much, transfer learning becomes harmful — the model gets "led astray" into learning inapplicable distance mappings.
The researchers specifically note: moderate out-of-distribution tests mostly benefit from transfer learning, but extreme out-of-distribution tests are hurt by it. This suggests transfer learning has a "sweet spot" — the auxiliary and main tasks' distance distributions should overlap, but not coincide completely.
What Does This Mean?
The findings have direct implications for Transformer architecture design:
The role of positional encodings needs re-examination. Mainstream LLMs almost universally use RoPE, but this research suggests RoPE may be a bottleneck for distance generalization. If a model must handle token dependency distances unseen at training time (extremely common in practice), RoPE's hard-coded distance bias becomes a limitation.
NoPE is not "no positional encoding" — it's "another kind of positional encoding". NoPE relies on implicit positional signals from causal attention, which is itself an encoding scheme — except what it learns is not "how to handle distance = 15" but "how to infer distance from the data." The latter is more flexible and transferable.
Data diversity is not a panacea. Increasing distance coverage in training helps, but marginal returns diminish quickly. What matters more is letting the model understand that "distance is variable," rather than exposing it to every distance — the latter is neither realistic nor necessary.
Transfer learning has a sweet spot. When choosing auxiliary tasks, the overlap of distance distributions is the key variable. Too close adds nothing; too far causes harm.
Limitations
This research uses synthetic tasks (copying, selective copying), which differ from real language tasks. "Distance dependencies" in real language are far more complex than copying — semantic, syntactic, and pragmatic dependencies are intertwined. Whether NoPE's advantage on synthetic tasks transfers to real language tasks still needs verification.
The researchers also acknowledge their experiments are small-scale (small Transformers), and behavior on large models may differ. But prior NoPE findings on length generalization have already been validated on large models, so the distance-generalization advantage of NoPE has a decent chance of scaling up too.
My Take
What interests me most about this paper is not the "NoPE is better" conclusion itself, but the tension it reveals between architectural bias and generalization ability. Positional encoding is an inductive bias — you tell the model "position information matters, please handle it specially." This bias works within the training distribution but becomes shackles outside it.
This echoes a classic deep learning paradox: strong inductive biases help with little data and hurt with lots of data. Convolution's translation invariance, recurrence's temporal assumptions, attention's permutation invariance — all the same class of problem. NoPE's victory is essentially the victory of "letting the data speak": when you don't impose distance assumptions on the model, it learns a more flexible distance-handling ability from the data.
Positional encodings won't disappear — their advantages in training efficiency and inference stability are too obvious. But this research reminds us: components we consider indispensable may just be artifacts of training convenience. Next time you see a model underperforming on long-context tasks, perhaps the question isn't "how to improve the positional encoding" but "is the positional encoding the problem itself."
---
Paper: Distance generalization in transformers: why bother with positional encoding?
Authors: Daniel Henrik Nevermann, Claudius Gros (Goethe University Frankfurt)
Published: 2026-09-10