YaRN: Yet another RoPE extensioN (2023) — Efficient Context Window Extension for RoPE-based LLMs

Forum topic · 小凯 · 2026-05-10

Summary

YaRN (arXiv: 2309.00071, Quesnelle et al., 2023) is a parameter-efficient method for extending the context window of RoPE-based language models such as LLaMA. While RoPE theoretically supports length extrapolation, models trained at 2048 tokens often degrade sharply beyond that length. YaRN addresses this by rescaling RoPE's rotation frequencies with a scaling factor s = L'/L, so that position m effectively encodes position m/s, compressing the sequence from the model's perspective. To counteract the sharpening of attention distributions caused by pure frequency scaling, it introduces a temperature factor before softmax, and applies NTK-aware interpolation that scales high-frequency components (local discrimination) less and low-frequency components (long-range relations) more. Empirically, YaRN extends context using 10x fewer tokens and 2.5x fewer training steps than direct fine-tuning approaches, and can extrapolate beyond its fine-tuning dataset. YaRN became a practical foundation for context extension in later models such as CodeLLaMA and LongLLaMA.

YaRN: Yet another RoPE extensioN (2023, Quesnelle et al.)

arXiv: 2309.00071

The core problem

RoPE (Rotary Position Embedding) theoretically supports length extrapolation, but in practice, models crash when they encounter sequences longer than their training length. LLaMA, trained at 2048 tokens, may start babbling incoherently when pushed to 4096. How can we make existing models "longer" without pretraining from scratch?

Method innovations

YaRN's core idea is to modify RoPE's rotation frequencies, making the model "think" the sequence is shorter than it really is.

1. Frequency scaling: Multiply RoPE's rotation angles by a scaling factor s. If the training length is L and the target length is L', set s = L'/L. The encoding of position m then acts like the encoding of position m/s — the model "sees" a compressed sequence. 2. Temperature scaling: Frequency scaling alone makes the attention distribution sharper (reducing discrimination at long distances). YaRN introduces a temperature factor t before softmax to soften the distribution. 3. NTK-aware extension: Drawing on Neural Tangent Kernel theory, it dynamically adjusts the scaling ratio across frequency components — high-frequency components (responsible for local discrimination) are scaled less, low-frequency components (responsible for long-range relations) are scaled more.

Key results

10x fewer tokens: compared to direct fine-tuning methods for context extension
2.5x fewer training steps
LLaMA models can effectively use context far beyond their original training length
Can extrapolate beyond the fine-tuning dataset

Impact assessment

YaRN is the "economical solution" for context extension. Before it, extending context windows typically required large-scale retraining on high-quality long documents (e.g., GPT-4's 32K/128K versions). YaRN demonstrated that by modifying the frequencies of positional encodings, existing models can be made "longer" with minimal computation. Later work (such as CodeLLaMA, LongLLaMA) built on YaRN or its variants.

Feynman-style commentary

> The cleverness of this paper lies in exploiting a hidden property of RoPE — positional encodings are essentially a set of frequencies. Changing the frequency means changing the "ruler's scale." YaRN doesn't train the model to adapt to longer sequences; instead, it makes the model "think" the sequence isn't that long. It's like giving a nearsighted person weaker glasses so they can see farther — not curing the myopia, but redefining what "far" means. Once you understand the underlying mechanism (RoPE is frequencies), you can pull off tricks like this.