English static mirror for SEO/GEO · AI-assisted translation · Read Chinese original

Information Abundance Paradox: Longer Training Contexts Weaken a Model's Parametric Knowledge

Forum topic · ✨步子哥 · 2026-08-13

Summary

A 2026 paper by Arda Uzunoglu, Benjamin van Durme, and Daniel Khashabi introduces the "Information Abundance Paradox": the longer the context window used during language model training, the worse the model performs at independently recalling knowledge. Experiments on Phi-3 and OLMo 3 models, trained with identical token budgets but varying context lengths (4K to 128K), reveal an inverted U-curve: SuperGLUE and MCQA performance peaks near 2048 training tokens, while LM loss bottoms out near 8192 tokens, then degrades. The effect persists across model sizes from 20M to 750M parameters, so extra capacity does not fix it. The authors frame this as a shift from parametric internalization to contextualization—models that see abundant relevant text in context learn to rely on lookup instead of encoding facts into weights. Practical takeaways: optimal training context length depends on evaluation needs, and evaluations must cover both open-book and closed-book settings.

The Counterintuitive Finding

In August 2026, Arda Uzunoglu et al. published a paper titled *Information Abundance Paradox: Long-Context Training Undermines Parametric Knowledge*.

The core finding in one sentence: the longer the context window during training, the worse the model performs on tasks requiring independent knowledge recall.

This is not an engineering bug or a data quality issue—it is a structural phenomenon hidden for years behind the intuition that "longer context = better." The paper names it the Information Abundance Paradox.

A Everyday Analogy

Imagine preparing for an open-book exam with two study strategies:

  • Strategy A: memorize all the key content; only flip through the book once or twice to confirm
  • Strategy B: don't memorize; rely on the book throughout
  • Intuitively, an open-book exam should be easier. But if you know the book is always available, your brain quietly gets lazy—why memorize what's on the page? You perform well with the book in hand, but once it's taken away, you're lost.

    Long-context training gives language models a "giant open-book window." When training contexts contain enough relevant text, the model learns that "the information is in the context anyway, so I don't need to encode it into my parameters," and switches from memorize-mode to lookup-mode.

    The core mechanism: the richer the information in the training context, the weaker the incentive to internalize it into parameters.

    The Inverted U-Curve

    The experimental design is clean. Using Phi-3 and OLMo 3 models, with identical token budgets, the authors vary only the training context length (4K, 8K, 65K, 128K), then evaluate on multiple benchmarks.

    The result is a clean inverted U-curve:

  • SuperGLUE and MCQA: performance peaks around 2048 tokens, then declines as context grows
  • Language Modeling loss: reaches a minimum around 8192 tokens, then rises
  • In other words, models trained at 4K and 8K context systematically outperform 65K and 128K models on tasks requiring independent recall. This is not noise—the inflection point is statistically significant across multiple evaluation suites and model scales.

    Crucially: scaling up model capacity does not rescue it. The same inverted U pattern holds across four sizes from 20M to 750M parameters. This is not "the model is too small for long context"—long context changes the model's learning mode itself.

    Why an Inverted U, Not Monotonic Decline

    An inverted U means very short context is also bad. That makes sense: with too-short contexts, the model never sees enough contextual dependencies to learn long-range structure. But past a certain point, extending context starts to hurt.

    The inflection point depends on the length distribution of evaluation tasks. The paper finds the optimal training context for SuperGLUE/MCQA is shorter than for LM loss, because those benchmarks' evaluation instances are themselves shorter. In other words, the optimal training context length is tied to the length needed at evaluation time.

    Practical implication: it's not "longer is better" or "shorter is better"—you should find a middle value matched to your evaluation scenario.

    Mechanism: From Parametric to Contextual

    The paper's theoretical framework distinguishes two learning modes:

  • Parametric internalization: encoding information into weights, independent of context at inference
  • Contextualization: not encoding into weights, retrieving from context at inference
  • Long-context training pushes models toward the latter. This is not a bug—it follows from the optimization objective: if relevant information is always in context, "lazily" skipping parameter encoding yields lower loss.

    The problem: when the context lacks relevant information at evaluation (knowledge QA, few-shot evaluation), the model loses its independent recall ability. It learned to "use the book," but the book was taken away at exam time.

    Where This Paper Fits

    The paper challenges a widely accepted implicit assumption: that long-context training is a neutral scaling axis where longer is always better.

    The community knew long-context *inference* was hard (attention sparsity, positional extrapolation), but assumed extending training context only brings gains. When long-context training underperformed, the standard diagnosis was "not enough high-quality long documents," and the remedy was "find more long documents."

    This paper says: no—the problem is not data quantity, but the training mechanism itself. Long context changes the model's learning mode, and that change is not free.

    Relation to "Evaluation Blind Spot Laws"

    This paper is another instance of evaluation blind spots. Earlier blind spots involved metrics (loss, MMLU) masking critical failure modes. Here, the blind spot is the evaluation paradigm (open-book vs. closed-book) masking a switch in learning mode.

    If you only evaluate with context available, long-context training looks fine. Only closed-book settings (few-shot, zero-shot knowledge QA) reveal the degradation of parametric knowledge. This is structurally isomorphic to Progressive Cramming (99% token accuracy masking 100% generation failure): a single evaluation perspective can mask fundamental changes in learning mode.

    Practical Implications

    1. Long context is not free. Extending training context changes the learning mode; there is a trade-off between context-use ability and independent knowledge recall. 2. An optimal training context length exists. Not longer-is-better—match it to your evaluation scenario. 3. Evaluate both modes. Include both open-book and closed-book evaluations to see the full picture of learning modes. 4. More capacity doesn't fix it. The inverted U holds from 20M to 750M parameters—this is a training mechanism issue, not a capacity issue.

    A Deeper Observation

    The paradox suggests a more general principle: capability is not a single dimension but a composition of multiple modes. A model can be good at "using the book" or good at "memorizing the book"—two different abilities. Long-context training strengthens the former while weakening the latter. This is not a mere trade-off but a mode shift: the training environment reshapes the model's learning mode.

    A growing cross-paper consensus: the structure of the training environment determines the structure of the model's capabilities.

    Paper Information

  • Title: Information Abundance Paradox: Long-Context Training Undermines Parametric Knowledge
  • Authors: Arda Uzunoglu, Benjamin van Durme, Daniel Khashabi
  • arXiv: https://arxiv.org/abs/2608.12218
  • Code: https://github.com/ardauzunoglu/information-abundance-paradox

Tags

#long-context#language-models#parametric-knowledge#information-abundance-paradox#training-dynamics#evaluation#inverted-u-curve#machine-learning-research

This page is an English static mirror generated for search and AI citation. It may be a full translation or structured summary of the Chinese original. Canonical interactive discussion lives on the Chinese page: https://zhichai.net/topic/178633431