Randomized YaRN: Teaching LLMs to Read Long Documents with Randomized Position Offsets

This post is a detailed Chinese forum walkthrough of the paper *"Randomized YaRN Improves Length Generalization for Long-Context Reasoning"* (Mehta, Yin & Durrett, arXiv:2606.23687), explaining how to cure the "long-range myopia" of LLMs trained on short contexts.

The Problem: Long-Range Myopia

LLMs are typically pretrained on 4K–8K token contexts. When asked to process 100K+ token documents—technical manuals, legal contracts, entire books—their performance collapses, like a mystery reader who forgets the clues from page one by the time the detective reveals the killer.

Background: Positional Encoding and Context Extension

The post reviews the evolution of positional encoding:

• Absolute positions — simple integers; unstable for large values.
• Sinusoidal encodings — wave patterns of different frequencies giving each position a unique fingerprint.
• RoPE (Rotary Position Embedding) (arXiv:2104.09864) — encodes position as rotations in a complex plane, making relative position naturally visible in attention scores.

Because attention is O(n²), training long sequences is expensive, so models are trained short and then extended via extrapolation methods:

1. Naive extrapolation — fails catastrophically beyond the trained length. 2. Positional Interpolation (PI) (arXiv:2306.15595) — squeezes larger positions into the trained range; distant tokens become hard to distinguish. 3. NTK-aware interpolation — compresses low-frequency components more, preserving fine-grained high-frequency information. 4. YaRN (arXiv:2309.00071) — identifies attention entropy collapse in long sequences and fixes it with temperature scaling, smoothing the attention distribution.

The Core Idea of Randomized YaRN

The key question: if the model only sees positions 0–8192 during training, why not let it "preview" farther positions during training itself?

The method combines three components:

1. YaRN-style positional extension — position encodings cover a much larger range (e.g., 128K) than the training data (e.g., <8K). 2. Randomized position assignment — instead of always placing a sequence at positions [0, L-1], sample a random starting offset s and use [s, s+L-1]. A 2048-token sequence might occupy positions 45000–47047. 3. Length curriculum — gradually increase the randomization range across training stages (e.g., 8K → 32K → 128K).

Why It Works

RoPE is periodic: because of sin/cos functions, relative position relationships matter far more than absolute positions. Randomization exposes the model to diverse relative-distance patterns during short-sequence training—nearby tokens may be 10 apart, or 50,000 apart in absolute index. It's like a piano student who practices around middle C but occasionally touches the high register: the interval relationships transfer.

Randomization also changes the positional distribution: no position is always the "beginning," and the model learns genuinely relative positions.

Mathematical Formulation

• RoPE attention: f(q, m, k, n) = q^T · R(θ_m) · R(θ_n)^T · k, with θ_i = 10000^(-2i/d).
• YaRN temperature scaling: Attention(Q, K, V) = softmax(QK^T / (t · √d_k)) · V, with t typically 0.75–1.
• Randomized sampling: for sequence length L, start position s ~ Uniform from [0, L_max], where L_max >> L.

Experimental Results

Tested on two long-context reasoning benchmarks:

• BABILong (arXiv:2406.11249) — retrieving facts buried in extremely long, distractor-filled documents. With training data under 8K: standard fine-tuning degrades at 32K and nearly fails at 128K; Randomized YaRN significantly outperforms it at all lengths from 16K to 128K, with the largest gains (tens of percentage points) at the most out-of-distribution lengths.
• MRCR — tracking entity coreference across many conversation turns; Randomized YaRN shows substantial improvements, especially as conversations extend to dozens or hundreds of turns.

Ablations

• Pure YaRN (no randomization) helps but underperforms Randomized YaRN — randomization adds regularization.
• Randomization range matters: too small (16K) is ineffective, too large (1M) is unstable; 128K is a sweet spot.
• The length curriculum is necessary for stable, best-performing training.

Significance and Open Questions

• Training efficiency: no long documents required—long-context generalization emerges from short-sequence data plus randomization. This is especially valuable for the open-source community with limited compute.
• Applications: legal contract review, full medical histories, whole-paper scientific QA, codebase-level understanding.
• Open problems: performance beyond 1M tokens is untested; the optimal randomization distribution and curriculum pacing lack theoretical grounding; compatibility with sparse attention, memory mechanisms, and non-Transformer architectures (Mamba, RWKV) remains unexplored. Future directions include adaptive randomization and combining native long context with RAG.

References

• Mehta, M., Yin, F., & Durrett, G. (2026). Randomized YaRN Improves Length Generalization for Long-Context Reasoning. arXiv:2606.23687.
• Peng, B., et al. (2023). YaRN: Efficient Context Window Extension of Large Language Models. arXiv:2309.00071.
• Su, J., et al. (2021). RoFormer: Enhanced Transformer with Rotary Position Embedding. arXiv:2104.09864.
• Chen, S., et al. (2023). Extending Context Window of Large Language Models via Positional Interpolation. arXiv:2306.15595.
• Kuznetsov, I., et al. (2024). BABILong: Testing the Limits of LLMs on Long Context Reasoning. arXiv:2406.11249.
• bloc97 (2023). NTK-Aware Scaled RoPE. https://www.reddit.com/r/LocalLLaMA/comments/14lz7j5/