> *StepPO: Step-Aligned Policy Optimization for Agentic Reinforcement Learning* > arXiv: 2604.18401 | 2026
An Overlooked Misalignment
Today's AI agents—OpenClaw, Claude Code, Devin—can write code, operate browsers, manage file systems, and autonomously complete complex multi-step tasks. But the training methods behind them contain a fundamental misalignment.
Current RL methods for training LLMs (such as PPO and GRPO) model and optimize at the token level. Each token is an action; each token yields a reward signal. That's fine for chat—but agents don't chat. Agents execute steps: search the web, read a file, write code, run tests, fix a bug. A single step may contain dozens or hundreds of tokens, while the reward usually arrives only after the whole step completes.
Training step-level behavior with token-level optimization is like timing a marathon with a stopwatch calibrated for seconds—you're measuring something, just not the right thing.
StepPO: Upgrading the Unit of Optimization from Token to Step
StepPO's core claim is simple: RL for agents should take the step—not the token—as the basic unit. It achieves step-alignment on three levels:
- Step-level MDP: A traditional token-level MDP becomes a step-level MDP. An action is no longer a token but an entire step (e.g., one tool call and its result). State transitions describe how the environment changes after executing a step, not what the next token is.
- Step-level credit assignment (the key innovation): In token-level RL, if a step contains 50 tokens, the reward must be distributed across all 50—which contributed most? That's nearly impossible to judge accurately. StepPO assigns credit at the step level: the whole step shares one reward, with no need to split it across tokens.
- Step-level replay buffer: Training data is stored and sampled as complete steps rather than token sequences, avoiding the re-tokenization problem—the same step may tokenize differently in different contexts, but as a step it is stable.
Preliminary Experiments
The paper reports initial experiments on Agent-R1 and Claw-R1 (built on OpenClaw). Step-level PPO significantly outperforms token-level PPO on agent tasks—not surprising: when the objective shifts from "getting each word right" to "getting each step right," the model focuses on step-level decision quality rather than wording details.
Why It Matters
StepPO is more than a technical tweak; it represents a mindset shift from language model to decision model. When we use an LLM as an agent, it is no longer a machine that talks but a machine that decides—training should reflect that.
My Take
This is a position paper: its value lies in raising the problem and framing it, not delivering final answers. Agent RL currently carries a hidden assumption: since agents are built on LLMs, LLM RL methods should suffice. StepPO argues this is flawed—an LLM substrate does not mean the optimization granularity must be token-level. Cars and bicycles both have wheels, but you shouldn't repair a car with a bicycle manual.
Paper: arxiv.org/abs/2604.18401 Code: github.com/AgentR1/Agent-R1