[论文] KnowGPO: Knowledge-Grouped Policy Optimization for Search Agents

论文概要

研究领域: ML
作者: Siqi Chen, Zhengxi Lu, Yuchen Yan, Xingyu Wu, Qipeng Chen, Xin Zhang, Aiting Liu, Chao Deng, Jie Liu, Jin Ma, Weiming Lu, Jun Xiao, Yongliang Shen
发布时间: 2026-10-15
arXiv: 2610.22331

中文摘要

深度搜索智能体通过长时程多轮交互解决知识密集型任务，将信用分配给单个搜索决策从根本上是困难的。基于结果的强化学习仅在整条轨迹上共享一个终端奖励，而最近的过程监督方法将信用锚定在表面轨迹状态或模型内部信号上，在深度搜索中这两者都变得不可靠。我们提出 KnowGPO，通过轨迹所获取的任务相关知识而非表面观测来对齐轨迹。对于每个查询，KnowGPO 将推导答案所需的信息分解为原子知识条件，并通过已满足条件的子集表示轨迹状态。连续步骤之间的增量知识增益作为环境接地的过程奖励。在 Xbench、BrowseComp-ZH 和 GAIA 上，KnowGPO 持续优于基线 3.8 到 4.2 分，我们的 8B 模型可与大多数 32B 开源系统媲美。

原文摘要

Deep search agents solve knowledge-intensive tasks through long-horizon, multi-turn interactions, where assigning credit to individual search decisions is fundamentally difficult. Outcome-based reinforcement learning provides only a single terminal reward shared across an entire trajectory. Recent process-supervision methods aim to assign credit at finer granularity, but they typically anchor credit to either surface-level trajectory states or model-internal signals, both of which become unreliable in deep search where observations are heterogeneous and retrieved contexts are long and noisy. We propose KnowGPO, a reinforcement learning framework that aligns trajectories by the task-relevant knowledge they have acquired rather than by their surface observations. For each query, KnowGPO deco...

自动采集于 2026-10-17

#论文 #arXiv #ML #小凯