LongTraceRL: Learning Long-Context Reasoning from Search Agent Traces with Hierarchical Distractors and Rubric Rewards

Forum topic · 小凯 · 2026-06-02

Summary

LongTraceRL is a reinforcement learning framework for improving long-context reasoning in large language models, proposed by Nianyi Lin, Jiajie Zhang, Lei Hou, and Juanzi Li (arXiv:2605.31584). Long-context reasoning remains difficult because models struggle to locate and integrate key information amid extensive distracting content. While reinforcement learning with verifiable rewards (RLVR) shows promise, existing approaches are limited by low-confusability distractors and sparse outcome-based reward signals. LongTraceRL addresses both issues. For data construction, the authors generate multi-hop questions via random walks over knowledge graphs and use search agent trajectories to build hierarchical distractors: documents the agent read but did not cite (high confusability) and documents never opened from search results (low confusability). For reward design, they introduce Rubric Reward, which uses gold entities along each reasoning chain as fine-grained entity-level process supervision, applied only to responses with correct answers (positive policy). Across three reasoning LLMs (4B–30B parameters) and five long-context benchmarks, LongTraceRL consistently outperforms strong baselines, encouraging comprehensive, evidence-based reasoning.

Overview

Field: NLP / AI / ML Authors: Nianyi Lin, Jiajie Zhang, Lei Hou, Juanzi Li arXiv: 2605.31584 | PDF

Key Contributions

Long-context reasoning remains a core challenge for large language models: they often fail to locate and integrate critical information buried in large amounts of distracting content. Reinforcement learning with verifiable rewards (RLVR) shows promise for this task, but existing methods are constrained by low-confusability distractors and sparse, outcome-oriented reward signals. LongTraceRL tackles both limitations.

Hierarchical Distractor Construction

Multi-hop questions are generated via random walks over knowledge graphs.
Search agent trajectories are used to build hierarchical distractors:
Documents the agent read but did not cite (high confusability)
Documents from search results that were never opened (low confusability)

Rubric Reward

Uses gold entities along each reasoning chain as fine-grained, entity-level process supervision.
Applied only to responses with correct answers (positive policy).

Results

Evaluated on three reasoning LLMs (4B–30B) and five long-context benchmarks, LongTraceRL consistently outperforms strong baselines, encouraging comprehensive, evidence-based reasoning.

---

*Auto-collected on 2026-06-02.*