KnowGPO: Knowledge-Grouped Policy Optimization for Deep Search Agents

Forum topic · 小凯 · 2026-10-17

Summary

KnowGPO is a reinforcement learning framework for deep search agents that addresses credit assignment in long-horizon, multi-turn knowledge-intensive tasks. Unlike outcome-based RL, which provides only a single terminal reward per trajectory, and prior process-supervision methods that anchor credit to surface-level trajectory states or model-internal signals, KnowGPO aligns trajectories by the task-relevant knowledge they acquire. For each query, it decomposes the information needed to derive the answer into atomic knowledge conditions and represents trajectory state by the subset of conditions satisfied, treating incremental knowledge gain between consecutive steps as an environment-grounded process reward. On Xbench, BrowseComp-ZH, and GAIA benchmarks, KnowGPO consistently outperforms baselines by 3.8 to 4.2 points, with its 8B model matching most 32B open-source systems. Paper: arXiv 2610.22331.

Paper Overview

Field: ML Authors: Siqi Chen, Zhengxi Lu, Yuchen Yan, Xingyu Wu, Qipeng Chen, Xin Zhang, Aiting Liu, Chao Deng, Jie Liu, Jin Ma, Weiming Lu, Jun Xiao, Yongliang Shen Published: 2026-10-15 arXiv: 2610.22331

Abstract

Deep search agents solve knowledge-intensive tasks through long-horizon, multi-turn interactions, where assigning credit to individual search decisions is fundamentally difficult. Outcome-based reinforcement learning provides only a single terminal reward shared across an entire trajectory. Recent process-supervision methods aim to assign credit at finer granularity, but they typically anchor credit to either surface-level trajectory states or model-internal signals, both of which become unreliable in deep search where observations are heterogeneous and retrieved contexts are long and noisy.

Key Idea

KnowGPO is a reinforcement learning framework that aligns trajectories by the task-relevant knowledge they have acquired rather than by their surface observations. For each query, KnowGPO decomposes the information required to derive the answer into atomic knowledge conditions and represents trajectory state by the subset of conditions satisfied. The incremental knowledge gain between consecutive steps serves as an environment-grounded process reward.

Results

Evaluated on Xbench, BrowseComp-ZH, and GAIA.
KnowGPO consistently outperforms baselines by 3.8 to 4.2 points.
The 8B model is competitive with most 32B open-source systems.

---

*Auto-collected on 2026-10-17*