Copy Less, Ground More: Fixing Repetitive Copying in Long-Context LLM Reasoning with GEAR

Forum topic · 小凯 · 2026-07-23

Summary

This paper identifies a critical failure mode in long-context LLM reasoning called repetitive copying, where models extensively copy input text into their reasoning traces instead of solving problems productively. The authors show this behavior is pervasive across frontier long-context models and intensifies with context length. By decomposing prompts into task-relevant key evidence and irrelevant distractor context, they trace the root cause to insufficient grounding: models copy indiscriminately, and those failing to focus on key evidence answer incorrectly far more often. They propose GEAR (Grounding Evidence-Aware Reward), a reward-shaping method that grants grounding rewards for overlap with key evidence and distractor penalties for overlap with irrelevant context, plus an automated pipeline to build evidence-annotated training data from arbitrary documents. Across multiple model scales and benchmarks, GEAR improves over accuracy-only RL by up to +4.6 points on average, with larger gains at longer contexts, while reducing repetitive copying and thinking length. Paper: arXiv 2507.17091.

Paper Overview

Field: NLP Authors: Lizhe Fang, Weizhou Shen, Tianyi Tang Published: 2026-07-22 arXiv: 2507.17091

Key Points

New failure mode — repetitive copying: Long-context LLMs that generate step-by-step reasoning traces extensively copy text from the input into their reasoning instead of productively solving the problem. This behavior is pervasive across frontier long-context models and intensifies as context length grows.
Root cause: insufficient grounding: By decomposing each prompt into task-relevant key evidence and irrelevant distractor context, the authors show that models copy from the prompt indiscriminately. Models that fail to focus on key evidence are far more likely to answer incorrectly.
GEAR (Grounding Evidence-Aware Reward): A reward-shaping method for reinforcement learning that provides:
*Grounding rewards* for overlap with key evidence in the reasoning trace
*Distractor penalties* for overlap with irrelevant context
Automated data pipeline: To apply GEAR on natural-language data, the authors develop a pipeline that automatically builds evidence-annotated training data from arbitrary documents.
Results: Evaluated across multiple model scales and benchmarks, GEAR improves over standard accuracy-based RL by up to +4.6 points on average, with larger gains at longer context lengths, while reducing repetitive copying and thinking length.

Takeaway

Even as long-context evaluation shifts from simple retrieval to complex reasoning, accurately grounding reasoning in relevant evidence remains an essential and highly improvable capability.

---

*Auto-collected on 2026-07-23*