LongTraceRL: Learning Long-Context Reasoning from Search Agent Trajectories with Rubric Reward

Forum topic · 小凯 · 2026-06-02

Summary

LongTraceRL is a reinforcement learning framework for improving long-context reasoning in large language models, addressing the core problem that LLMs struggle to locate and integrate key information amid extensive distractors. While reinforcement learning with verifiable rewards (RLVR) shows promise for this task, existing approaches are limited by low-confusability distractors and sparse outcome-based reward signals. The authors construct training data by generating multi-hop questions via random walks over knowledge graphs, then use search agent trajectories to build hierarchical distractors: documents the agent read but did not cite (high confusability) versus documents never opened from search results (low confusability). For reward design, they propose a Rubric Reward that uses golden entities along each reasoning chain as fine-grained entity-level process supervision, applied only to responses with correct answers (positive policy). Experiments across three reasoning LLMs (4B-30B) and five long-context benchmarks show LongTraceRL consistently outperforms strong baselines and encourages comprehensive, evidence-based reasoning. Paper: arXiv 2605.31584 by Nianyi Lin, Jiajie Zhang, Lei Hou, and Juanzi Li.

Paper Overview

Field: NLP/AI/ML Authors: Nianyi Lin, Jiajie Zhang, Lei Hou, Juanzi Li Published: 2026-05-29 arXiv: 2605.31584 PDF: 2605.31584.pdf

Abstract

Long-context reasoning remains a core challenge for large language models, which often struggle to locate and integrate key information amid large amounts of distractor content. Reinforcement learning with verifiable rewards (RLVR) shows promise for this task, but existing methods are limited by low-confusability distractors and sparse outcome-oriented reward signals.

This post summarizes LongTraceRL, a framework addressing both limitations:

Data Construction

Multi-hop questions are generated via random walks over knowledge graphs.
Search agent trajectories are used to build hierarchical distractors:
Documents the agent read but did not cite — high confusability.
Documents in search results that were never opened — low confusability.

Reward Design

A Rubric Reward uses golden entities along each reasoning chain as fine-grained, entity-level process supervision.
The reward is applied only to responses with correct answers (positive policy).

Results

Evaluated on three reasoning LLMs (4B–30B) and five long-context benchmarks.
LongTraceRL consistently outperforms strong baselines, encouraging comprehensive, evidence-based reasoning.

--- *Auto-collected on 2026-06-02*