SPO++: Stream-Aligned Policy Optimization for Asynchronous Agentic RL

Forum topic · 小凯 · 2026-08-27

Summary

SPO++ is a new reinforcement learning method for asynchronous agentic RL presented in arXiv paper 2608.24870 by Kai Ruan and colleagues. Group-relative reinforcement learning requires waiting for sibling rollouts of the same prompt, which is costly for long, variable tool-use trajectories. Single-stream Policy Optimization (SPO) removes this dependency using a persistent prompt-level value estimate, but whitens one advantage per trajectory before optimizing a token-mean actor loss. The authors show that trajectory centering generally does not center the token-weighted quantity the actor actually consumes. SPO++ fixes this mismatch by standardizing terminal-outcome advantages under the action-token measure, and additionally organizes prompt evidence by the policy event that generated it rather than learner receipt order. Across matched runs on ALFWorld at two model scales and on Math-TIR, SPO++ improves online learning efficiency over SPO. Pairwise ablations identify action-token-measure normalization as the strongest tested component.

Overview

Field: AI
Authors: Kai Ruan, Jinghao Lin, Qianshan Wei, Ziqi Zhou, Zihe Huang
Published: 2026-08-25
arXiv: 2608.24870

Abstract

Group-relative reinforcement learning waits for sibling rollouts of the same prompt, which is costly for long and variable tool-use trajectories. Single-stream Policy Optimization (SPO) removes this dependency with a persistent prompt-level value estimate, but its recipe whitens one advantage per trajectory before optimizing a token-mean actor loss.

The authors show that trajectory centering generally does not center the token-weighted quantity consumed by the actor, and fix this mismatch by standardizing terminal-outcome advantages under the action-token measure. Additionally, SPO++ organizes prompt evidence by the policy event that generated it rather than by learner receipt order.

Results

Across matched runs on ALFWorld at two model scales and on Math-TIR, SPO++ improves online learning efficiency over SPO. Pairwise ablation experiments identify action-token-measure normalization as the strongest tested component.

*Auto-collected on 2026-08-27.*