Paper Overview
Field: Machine Learning Authors: Haofei Yu, Yixing Chen, Yining Zhao, Xuyan Ye, Yuxiang Liu Published: 2026-10-14 arXiv: 2610.20597
Summary
Training language agents is bottlenecked by the high cost of long-horizon rollouts, yet standard agentic reinforcement learning (RL) learns only from a terminal scalar reward, leaving the supervision in intermediate steps largely unused.
Prior work exploits this supervision through two types of targets, both hard to construct well:
1. Value targets — estimate per-step success probabilities, but derive from the same single scalar signal for every step in a trajectory. 2. Observation targets — fit environment responses (e.g., web pages and terminal logs), but these are long, noisy, and reach only one step ahead.
Method: CoPE
CoPE adds a third kind of supervision: multi-step action targets.
- On successful on-policy trajectories, CoPE trains the policy backbone to generate the action sequences following each history.
- Every action is additionally supervised when predicted from its preceding history, so the shared backbone absorbs a temporal ensemble over the course of training.
- Acting and forecasting thus co-evolve during training.
- At inference time, the policy still acts step by step — no extra inference overhead.
- arXiv: <https://arxiv.org/abs/2610.20597>
Results
Evaluated on five benchmarks — ALFWorld, ScienceWorld, WebShop, tau2-bench, and AppWorld — CoPE:
1. Achieves stronger overall performance than GRPO and RL baselines with prediction-auxiliary objectives in simulated environments, while also improving overall results on two real-world applications. 2. Achieves up to 1.9x sample efficiency over ECHO and 2.0x over GRPO on ScienceWorld. 3. Delivers these gains with a single policy backbone — no additional models, no separate pretraining, and no inference overhead.