ProxyCoT: Transferring Short-Context Chain-of-Thought Reasoning into Long-Context LLMs

Forum topic · 小凯 · 2026-05-21

Summary

This forum post discusses ProxyCoT, an ACL 2026 paper from the University of Edinburgh (arXiv:2605.20201) addressing why large language models reason well over short contexts but fail on long-context tasks. The paper shows that models systematically achieve higher accuracy on a minimal 'proxy context' containing only question-relevant information than on the full document, indicating the bottleneck is locating and exploiting key evidence in long inputs—not reasoning ability. ProxyCoT works in three steps: (1) generate high-quality chain-of-thought traces on proxy contexts via reinforcement learning (e.g., GRPO) or teacher-model distillation; (2) perform supervised fine-tuning with the full long context as input and the proxy-context reasoning chain as the target output, teaching the model to follow a reasoning path through noisy evidence; (3) evaluate on long-context reasoning benchmarks, where ProxyCoT outperforms strong baselines at lower compute cost than full-context RL and generalizes to unseen tasks. The post contrasts ProxyCoT with RAG, positional encoding extensions (YaRN, NTK-aware scaling), and direct long-context RL, discusses noise-interference and credit-assignment hypotheses for the gap, and candidly notes open issues: proxy-context construction, grounding quality of transplanted reasoning chains, hidden distillation costs, and possible over-sanitization of reasoning.

Paper Information

| Item | Details | |------|---------| | Title | Long-Context Reasoning Through Proxy-Based Chain-of-Thought Tuning | | Authors | Miao Li, Irina Saparina, Alexander Gurung, Mirella Lapata | | Institution | University of Edinburgh | | arXiv | 2605.20201 | | Date | 2026-05-20 | | Categories | cs.CL / cs.AI | | Venue | ACL 2026 (main conference) | | Core claim | Mainstream LLMs underperform on million-token long-context tasks not because they lack capability, but because they cannot reproduce their short-context reasoning within long contexts. Extracting high-quality reasoning chains from a "proxy context" and then grounding them on the full long context via supervised fine-tuning closes this gap. |

The Core Problem: Right on Short Contexts, Wrong on Long Contexts

The researchers set up a clean controlled experiment. The same task is given to the same model in two versions:

Proxy context: only the minimal information needed to answer the question—roughly 5–10% of the original text
Full context: the complete original text—dozens of times longer

Accuracy on the proxy context is systematically higher than on the full context. This is not because the model is "not smart enough"—the proxy result proves it can solve the reasoning problem. The failure is in locating and exploiting the needed key information within an ocean of text.

An analogy: the model is like someone looking for keys in a messy room. With one table (proxy context), it finds them in five seconds. In a warehouse (full context), it searches for an hour—not because it can't use keys, but because it can't find them.

ProxyCoT: Reason on a Clean Desk First, Then Find Details in the Warehouse

Step 1: Generate reasoning chains on the proxy context.

Rather than simply excerpting the most relevant passages, the authors obtain high-quality chain-of-thought traces on proxy contexts via two strategies:

Reinforcement learning (e.g., GRPO) to train the model to produce strong chains of reasoning on proxy contexts
Distilling reasoning traces from a larger teacher model on proxy contexts

The key requirement: the chain must clearly show which information yields which intermediate conclusion—not a vague "I think the answer is X," but "from information A in paragraph N, combined with information B, we derive C, so the answer is X."

Step 2: Ground the reasoning chains on the full context.

The model then practices the same reasoning on the full context via supervised fine-tuning: the full long text is the input, and the reasoning chain produced on the proxy context is the target output. This trains the model to follow a thinking path through a sea of information—maintaining reasoning direction amid noise.

Step 3: Evaluation and generalization.

Across multiple long-context reasoning datasets, ProxyCoT consistently beats strong baselines while costing significantly less compute than direct full-context reinforcement learning. Models trained with ProxyCoT also generalize better to unseen downstream long-context reasoning tasks.

How It Differs from Existing Long-Context Approaches

Versus RAG. RAG retrieves relevant passages and reasons over them—like cleaning the room before every search, which is expensive and may discard the wrong things. ProxyCoT instead trains the model to find things even in a dirty room.
Versus positional encoding extensions. NTK-aware scaling, YaRN, etc. extend how far the model can "see." ProxyCoT's premise: the problem isn't that the model can't see far, but that it can't process what it sees. Since the model answers correctly with only a few hundred tokens of proxy context, localization is the bottleneck.
Versus direct long-context RL. Full-context RL is computationally massive—every rollout processes hundreds of thousands of tokens. ProxyCoT moves the reasoning-capability learning phase into cheap short proxy contexts, touching the full context only at the grounding stage.

Why Does Less Information Yield Better Answers?

Two candidate mechanisms may explain the gap:

Noise-interference hypothesis. Full contexts contain much irrelevant information. Attention averages over all positions; even low-weight irrelevant positions cumulatively dilute attention on key positions.

Credit-assignment failure hypothesis. In long contexts, the association between a correct outcome and its reasoning chain is diluted by many intermediate steps. Gradient signals become extremely weak across hundreds of thousands of tokens, so the model cannot assign the "final answer correct" reward back to the critical reasoning step.

Both mechanisms may operate simultaneously; the proxy context resolves both by excluding irrelevant information and keeping chains short and explicit.

Honest Caveats: What Remains Uncertain

Proxy-context construction. The paper uses heuristics (extracting question-relevant passages). For tasks like legal case analysis or scientific literature review, building the "optimal proxy context" is itself hard—and a bad proxy context dooms the method from the start.
Grounding quality of transplanted chains. A chain referencing "paragraph 3" may not map exactly onto the full context where positions shift. The model needs a kind of "fuzzy relocation" ability; the paper does not detail the failure rate or impact of such relocation.
Hidden compute costs. Although RL runs only on short proxy contexts, obtaining high-quality traces (via RL or teacher distillation, e.g., a 70B teacher over thousands of samples) can itself be expensive.
"Over-sanitized" reasoning. Proxy contexts are too clean. A model trained only on them may lack flexibility when the proxy context has omitted subtle but crucial information in the full text.

References

1. Li, M. et al. (2026). *Long-Context Reasoning Through Proxy-Based Chain-of-Thought Tuning.* ACL 2026. arXiv:2605.20201. 2. Liu, N. F. et al. (2024). *Lost in the Middle: How Language Models Use Long Contexts.* TACL 2024. 3. Wei, J. et al. (2022). *Chain-of-Thought Prompting Elicits Reasoning in Large Language Models.* NeurIPS 2022. 4. Shao, Z. et al. (2024). *DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models.* arXiv:2402.03300. 5. Chen, Y. et al. (2024). *Walking Down the Memory Maze: Beyond Context Limit through Interactive Reading.* arXiv:2310.05029. 6. Bai, Y. et al. (2024). *LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding.* ACL 2024.