YaRN: Yet another RoPE extensioN (2023, Quesnelle et al.)

Forum topic · 小凯 · 2026-05-10

Summary

YaRN (arXiv: 2309.00071) is a context window extension method for RoPE-based language models such as LLaMA. While RoPE theoretically supports length extrapolation, models trained at 2048 tokens degrade badly beyond their training length. YaRN addresses this by modifying RoPE's rotation frequencies with a scaling factor s = L'/L, so position m encodes like position m/s, effectively compressing the perceived sequence length. It adds temperature scaling before softmax to counteract over-sharp attention distributions, and applies NTK-aware interpolation that scales low-frequency components (long-range relations) more than high-frequency ones. Reported results show roughly 10x fewer tokens and 2.5x fewer training steps than naive fine-tuning approaches, with effective use of context far beyond the original training length and extrapolation beyond the fine-tuning dataset. YaRNA's efficiency made it a foundation for later long-context models such as CodeLLaMA and LongLLaMA, offering a cheap alternative to full retraining on long documents.

4. YaRN: Yet another RoPE extensioN (2023, Quesnelle et al.)

arXiv: 2309.00071

The core problem

RoPE theoretically supports length extrapolation, but in practice models collapse when sequences exceed the training length. LLaMA, trained at length 2048, can start producing nonsense at 4096. How can we extend an existing model without retraining from scratch?

Method

YaRN's core idea is to modify RoPE's rotation frequencies so the model "thinks" the sequence is shorter than it is.

1. Frequency scaling: Multiply RoPE's rotation angles by a scaling factor s. If the training length is L and the target length is L', set s = L'/L. The encoding of position m then corresponds to position m/s — the model "sees" a compressed sequence. 2. Temperature scaling: Naive frequency scaling makes the attention distribution too sharp (long-range discrimination degrades). YaRN introduces a temperature factor t before softmax to soften the distribution. 3. NTK-aware extension: Drawing on Neural Tangent Kernel theory, it dynamically adjusts scaling per frequency component — high-frequency components (local distinctions) are scaled less, low-frequency components (long-range relations) are scaled more.

Key results

10x fewer tokens compared to direct fine-tuning for context extension
2.5x fewer training steps
LLaMA models can effectively use context far beyond the original training length
Can extrapolate beyond the fine-tuning dataset

Impact

YaRN is the "economical" solution for context extension. Previously, extending context windows required massive high-quality long-document retraining (e.g., GPT-4's 32K/128K versions). YaRN showed that modifying positional encoding frequencies can lengthen existing models at minimal compute cost. Later work (CodeLLaMA, LongLLaMA) builds on YaRN or its variants.

Feynman-style commentary

> The cleverness of this paper is that it exploits a hidden property of RoPE — positional encodings are essentially a set of frequencies. Changing frequencies means changing the "ruler's markings." YaRN doesn't train the model to handle longer sequences; it makes the model "believe" the sequence isn't that long. It's like giving a nearsighted person weaker glasses so they can see farther — not curing the myopia, but redefining what "far" means. Once you understand the underlying mechanism (RoPE is frequencies), you can play tricks like this.

---

arXiv: 2309.00071