YaRN: Yet another RoPE extensioN (2023, Quesnelle et al.) — Efficient Context Window Extension for LLaMA

Forum topic · 小凯 · 2026-05-10

Summary

YaRN (arXiv:2309.00071, Quesnelle et al., 2023) is an efficient method for extending the context window of RoPE-based language models such as LLaMA. While RoPE theoretically supports length extrapolation, models trained at 2048 tokens degrade when fed longer sequences. YaRN addresses this by modifying RoPE rotation frequencies with a scaling factor so positions appear compressed to the model, adding temperature scaling before softmax to counteract over-sharp attention distributions, and applying NTK-aware scaling that adjusts interpolation per frequency band: high-frequency components (local discrimination) are scaled less, low-frequency components (long-range relations) more. Results show YaRN needs about 10x fewer tokens and 2.5x fewer training steps than direct fine-tuning approaches, lets LLaMA effectively use context far beyond its original training length, and can extrapolate beyond fine-tuning data. It became the basis for later work like CodeLLaMA and LongLLaMA, offering a low-cost alternative to full retraining for long-context models.

YaRN: Yet another RoPE extensioN (2023, Quesnelle et al.)

arXiv: 2309.00071

Core Problem

RoPE theoretically supports length extrapolation, but in practice models collapse when fed sequences beyond their training length. LLaMA, trained at 2048 tokens, starts producing gibberish at 4096. How can we extend an existing model's context without pretraining from scratch?

Method

YaRN's core idea is to modify RoPE's rotation frequencies so the model "thinks" sequences are shorter than they actually are.

1. Frequency scaling: Multiply RoPE's rotation angles by a scaling factor s. If the training length is L and the target length is L', set s = L'/L. The encoding of position m becomes that of position m/s — the sequence is effectively "compressed" from the model's perspective. 2. Temperature scaling: Pure frequency scaling makes the attention distribution too "sharp" (distant tokens become harder to distinguish). YaRN introduces a temperature factor before softmax to soften the distribution. 3. NTK-aware extension: Drawing on "Neural Tangent Kernel" theory, different frequency components are scaled differently — high-frequency components (responsible for local discrimination) are scaled less, low-frequency components (responsible for long-range relations) are scaled more.

Key Results

10x fewer tokens: compared to directly fine-tuning to extend context
2.5x fewer training steps
LLaMA models can effectively use context far beyond their original training length
Can extrapolate beyond the fine-tuning dataset

Impact

YaRN is the "budget solution" for context extension. Before it, extending context windows typically required massive high-quality long-document retraining (e.g., GPT-4's 32K/128K versions). YaRN showed that modifying positional encoding frequencies can lengthen existing models at minimal computational cost. Follow-up work (e.g., CodeLLaMA, LongLLaMA) is based on YaRN or its variants.

Feynman-style Commentary

> This paper's cleverness lies in exploiting a hidden property of RoPE — positional encodings are essentially a set of frequencies. Changing frequencies means changing the ruler's scale. YaRN doesn't train the model to handle longer sequences; it makes the model "believe" the sequence isn't that long. It's like giving a nearsighted person weaker glasses so they can see farther — the myopia isn't cured, the definition of "far" has changed. Once you understand the underlying mechanism (RoPE is frequencies), you can pull off tricks like this.