Beyond Context Length: Context Length Is Not the Only Difficulty — Three Hidden Dimensions of Long-Context Reasoning

*This post is a translation of a Chinese tech forum discussion of arXiv: 2610.22067.*

The Intuition Trap

Ask anyone working on long-context LLMs what makes a task hard, and most will answer without hesitation: "context length."

It's a natural intuition—a 128K-context task is obviously harder than a 4K one. So benchmarks compete on "how long a context can we sustain accuracy," with Needle-in-a-Haystack (NIAH) tests as the classic example: hide a needle in a very long text and see if the model can find it.

But this paper from the University of Edinburgh (arXiv: 2610.22067) says: length is only the surface; difficulty actually has three hidden dimensions.

Three Overlooked Difficulty Dimensions

Dimension 1: Evidence Scope

How much necessary information do you need to find?

A QA task might need just one passage (small scope), or might require integrating ten passages scattered across the text (large scope). At the same total context length, the latter is clearly harder.

Analogy: finding one chapter of a book in a library (small) vs. writing a literature review synthesizing twenty books (large). The reading amount may be similar, but the cognitive load is completely different.

Dimension 2: Evidence Dispersion

How spread out is the necessary information in the context?

Take 5 necessary passages: if they appear consecutively in one region, the model can read them in one pass; if they are evenly scattered across 100K tokens, the model must repeatedly jump, locate, and remember.

Analogy: a puzzle—5 pieces together are easy to grab; scattered across the living room, bedroom, kitchen, and garage, you must search the whole house. Same number of pieces, vastly different difficulty.

Dimension 3: Reasoning Depth

How many steps of dependent reasoning lead from evidence to answer?

Some questions are direct extraction (depth 1); others require inferring B from A, C from B, and the answer from C (depth 3). The longer the chain, the more likely an intermediate error corrupts the final answer.

Analogy: computing "2+3" vs. "((2+3)×4-5)/3"—same digits, but the latter needs multi-step reasoning where every step can fail.

The LURCH Benchmark: Making the Dimensions Quantifiable

The paper introduces LURCH (Long-context Understanding with Reasoning and Characterisation), containing 6,550 QA instances, each annotated with:

• The specific positions of supporting evidence
• A reasoning graph — the inference steps from evidence to answer

This makes all three dimensions precisely quantifiable rather than manually estimated. LURCH spans multiple domains, ensuring conclusions aren't domain-specific artifacts.

Three Counterintuitive Findings

Finding 1: At fixed length, all three dimensions strongly negatively correlate with performance

At the same context length, larger evidence scope, higher dispersion, and deeper reasoning all hurt performance. The effect sizes of the three dimensions are comparable to that of context length itself.

This means: contexts of identical length can differ in difficulty by orders of magnitude. Evaluating models by length alone severely overestimates performance on "seemingly equal-length but actually harder" tasks.

Finding 2: As length grows, dispersed-evidence tasks collapse

As context goes from short to long, degradation patterns diverge:

• Local-evidence tasks (evidence concentrated in one region): performance roughly stable
• Dispersed-evidence tasks (evidence scattered everywhere): sharp performance drops

This explains why NIAH is overly optimistic—the needle only needs to be found, not integrated. Real tasks often require extracting and combining information from multiple locations.

Finding 3: Difficulty compounds multiplicatively across dimensions

The most counterintuitive finding: tasks moderately high on all three dimensions are harder than tasks extreme on one dimension.

Intuition says "extreme on one axis" should be hardest. But the paper finds that moderately-high-everywhere tasks compound difficulty multiplicatively—the model must simultaneously handle dispersed evidence, broad search, and multi-step reasoning, and the combined burden is amplified, not merely added.

Analogy: running a marathon (one extreme dimension) vs. running a marathon while doing math and memorizing vocabulary (three moderate dimensions). The former is exhausting but requires one ability; the latter demands multiple cognitive resources at once, with higher total load.

Implications for Evaluation Methodology

1. Long-context evaluation cannot rely on length alone

Most benchmarks use context length as the primary axis (4K/32K/128K/1M), but difficulty varies enormously at the same length. Future evaluations should report the distribution across the three dimensions, not just length.

2. Needle-in-a-Haystack is not enough

NIAH tests only "finding information in long text," but real tasks require "find, integrate, reason." LURCH's three dimensions are closer to real scenarios.

3. Difficulty is three-dimensional, not one-dimensional

A model may excel at "large-scope evidence" tasks but struggle with "deep reasoning" tasks. Aggregate accuracy masks this imbalance. Reports should be broken down by dimension, like a medical checkup that never gives a single "health score."

Implications for Engineering Practice

For RAG (retrieval-augmented generation) or long-document QA systems:

1. Retrieval strategy should account for dispersion: if a query requires integrating multiple scattered evidence fragments, single-shot retrieval may not suffice—design multi-round retrieval targeting one fragment per round.

2. Context organization should account for evidence scope: clustering related evidence together reduces dispersion—more effective than increasing context length.

3. Reasoning steps should be explicit: for high reasoning depth, have the model write out intermediate steps (Chain-of-Thought) to reduce per-step error probability.

A Deeper Reflection: Difficulty Is Task Structure, Not Data Volume

The paper's most thought-provoking stance: difficulty is not a function of data volume but of task structure.

The same number of tokens, organized differently, can differ in difficulty by orders of magnitude—like the same pile of LEGO bricks assembled into a castle (structured) versus dumped in a heap (unstructured).

LLM evaluation has long been dominated by the "scale narrative"—more parameters, longer contexts, more data. But this paper reminds us: structure matters more than scale. A model that handles dispersed evidence and deep reasoning well may be more useful than one with a longer window that cannot handle these structures.

Limitations and Open Questions

1. The three dimensions may be incomplete: the paper acknowledges other possible dimensions, such as "evidence conflict" (contradicting evidence) or "evidence noise" (necessary information mixed with irrelevant information). The three dimensions are a choice, not an exhaustive list.

2. Reasoning graph annotation cost: LURCH requires manually annotated reasoning graphs, limiting scale. 6,550 instances is substantial, but compared to automatically generated NIAH, scalability is limited.

3. Model coverage: the paper tested multiple models but not the latest ultra-long-context models (e.g., Gemini 2M context). Whether those perform better on dispersed evidence remains to be validated.

Conclusion

While everyone chases longer context windows, this paper reminds us: length is not difficulty; structure is. A truly strong long-context model should not merely "read more," but "integrate across scattered information and stay accurate through multi-step reasoning."

Next time a model claims "1M-token context support," don't be immediately impressed. Ask three questions: How does it perform on dispersed evidence? On broad-scope search? On deep reasoning? Length is a marketing number; three-dimensional difficulty is real capability.

---

Paper link: https://arxiv.org/abs/2610.22067 Benchmark data: LURCH (detailed in Section 4 of the paper)