[论文] Co-Evolving Acting and Forecasting for Language Agents

论文概要

研究领域: ML 作者: Haofei Yu, Yixing Chen, Yining Zhao, Xuyan Ye, Yuxiang Liu 发布时间: 2026-10-14 arXiv: 2610.20597

中文摘要

训练语言智能体的瓶颈在于长时域rollout的高成本，而标准的智能体强化学习（RL）仅学习终端标量奖励，中间步骤的监督基本未被利用。先前工作通过估计每步成功概率的值目标或拟合环境响应的观测目标来利用这些监督，但两者都难以构建好：(1) 值目标来自轨迹中每步相同的单一标量信号；(2) 观测目标（如网页和终端日志）冗长且噪声大，且只能前进一步。我们提出CoPE，增加了第三类多步动作目标。在成功的on-policy轨迹上，CoPE训练策略骨干生成跟随每个历史的动作序列。每个动作在从前序历史预测时也受到监督，形成共享骨干在训练过程中吸收的时间集成，使行动和预测共同进化。推理时策略仍逐步行动，无额外开销。在五个基准（ALFWorld、ScienceWorld、WebShop、tau2-bench和AppWorld）上，CoPE：(1) 在模拟环境中取得比GRPO和带预测辅助目标的RL基线更强的综合性能，同时在两个真实应用上也提升了整体表现；(2) 在ScienceWorld上分别比ECHO和GRPO实现高达1.9倍和2.0倍的样本效率；(3) 仅用单一策略骨干即实现这些收益，无需额外模型、单独预训练或推理开销。

原文摘要

Training language agents is bottlenecked by the high cost of long-horizon rollouts, yet standard agentic reinforcement learning (RL) learns only from a terminal scalar reward, leaving the supervision in intermediate steps largely unused. Prior work exploits this supervision through value targets that estimate per-step success probabilities or observation targets that fit environment response. Both are hard to construct well: (1) value targets come from the same single scalar signal for each step in the trajectory, and (2) observation targets, such as web pages and terminal logs, are long and noisy and reach only one step ahead. We propose CoPE, which adds a third kind of multi-step action targets. On successful on-policy trajectories, CoPE trains the policy backbone to generate the action ...

#论文 #arXiv #ML #小凯