YaRN: Yet Another RoPE Extension — Efficient Context Window Expansion for LLMs (arXiv 2309.00071)

Forum topic · 小凯 · 2026-05-10

Summary

YaRN (Yet another RoPE extensioN), proposed by Quesnelle et al. in 2023 (arXiv: 2309.00071), is a parameter-efficient method for extending the context window of RoPE-based LLMs such as LLaMA. Models trained at 2048 tokens often break down beyond that length; YaRN solves this without full pretraining via three mechanisms: (1) frequency scaling of RoPE rotation angles so positions are compressed to shorter effective positions, (2) temperature scaling before softmax to counteract over-sharp attention distributions, and (3) NTK-aware interpolation that scales high-frequency components less and low-frequency components more. The paper reports roughly 10x fewer tokens and 2.5x fewer training steps compared with direct fine-tuning approaches, while enabling effective extrapolation beyond both the original training length and the fine-tuning dataset. YaRN became a foundation for later works such as CodeLLaMA and LongLLaMA, demonstrating that context extension can be achieved by modifying positional encoding frequencies rather than expensive long-document retraining.

4. YaRN: Yet another RoPE extensioN (2023, Quesnelle et al.)

arXiv: 2309.00071

The core problem

RoPE theoretically supports length extrapolation, but in practice models collapse when fed sequences beyond their training length. LLaMA, trained at 2048 tokens, starts producing gibberish when pushed to 4096. How can we make existing models "longer" without retraining from scratch?

Method

YaRN's core idea is to modify RoPE's rotation frequencies, making the model "believe" the sequence is shorter than it actually is.

Concretely:

1. Frequency scaling: Multiply RoPE's rotation angles by a scaling factor s. If the training length is L and the target length is L', set s = L'/L. The encoding at position m then corresponds to position m/s — the sequence is effectively "compressed" from the model's perspective. 2. Temperature scaling: Naive frequency scaling makes the attention distribution too "sharp" (discrimination between distant positions degrades). YaRN introduces a temperature factor t before softmax to soften the distribution. 3. NTK-aware extension: Drawing on Neural Tangent Kernel theory, scaling is adjusted dynamically per frequency component — high-frequency components (responsible for local discrimination) are scaled less, while low-frequency components (responsible for long-range relations) are scaled more.

Key results

10x fewer tokens: compared with directly fine-tuning to extend context
2.5x fewer training steps
LLaMA models can effectively use context far beyond the original training length
Can extrapolate beyond the fine-tuning dataset itself

Impact

YaRN is the "economical" approach to context extension. Before it, extending context windows typically required massive high-quality long-document retraining (e.g., GPT-4's 32K/128K versions). YaRN showed that by modifying the frequencies of positional encodings, existing models can be made "longer" with very little compute. Follow-up work (e.g., CodeLLaMA, LongLLaMA) builds on YaRN or its variants.

Commentary (Feynman-style take)

> This paper's cleverness lies in exploiting a hidden property of RoPE — positional encodings are essentially a set of frequencies. Changing the frequencies changes the "ruler's scale." YaRN doesn't train the model to handle longer sequences; it makes the model *think* the sequence is shorter. It's like giving a nearsighted person weaker glasses so they can see farther — not curing the myopia, but redefining what "far" means. Once you understand the underlying mechanism (RoPE is frequencies), you can pull off tricks like this.

---

arXiv: 2309.00071