StepPO: Agentic Reinforcement Learning Should Be Aligned by Steps, Not Tokens

Forum topic · 小凯 · 2026-04-21

Summary

StepPO (Step-Aligned Policy Optimization) is a position paper arguing that reinforcement learning for AI agents should use steps—not tokens—as the fundamental optimization unit. Current methods like PPO and GRPO perform token-level credit assignment, which the author compares to timing a marathon with a stopwatch: rewards for agent tasks typically arrive only after a multi-token action (such as a tool call) completes, making token-level attribution inaccurate. StepPO introduces three components: a step-level MDP where actions are whole steps like tool calls, step-level credit assignment where an entire step shares one reward signal, and a step-level replay buffer that avoids re-tokenization instability. Preliminary experiments on Agent-R1 and Claw-R1 (built on OpenClaw) show step-level PPO significantly outperforms token-level PPO on agentic tasks. The paper frames a conceptual shift from training language models to training decision-making models.

*StepPO: Step-Aligned Policy Optimization for Agentic Reinforcement Learning* arXiv: 2604.18401 | 2026

An Overlooked Misalignment

Modern AI agents—OpenClaw, Claude Code, Devin—can write code, operate browsers, manage file systems, and autonomously complete complex multi-step tasks. But the training methods behind them have a fundamental misalignment.

Current RL methods for LLMs (PPO, GRPO) model and optimize at the token level: every token is an action, every token generates a reward signal. That's fine for chat. But agents don't chat—they execute steps: search the web, read a file, write code, run tests, fix a bug. A step may span dozens or hundreds of tokens, yet the reward usually arrives only after the whole step completes.

Training step-level behavior with token-level optimization is like timing a marathon with a stopwatch—you're measuring something, but not the right thing.

StepPO: Upgrading the Optimization Unit from Token to Step

The paper's core claim is simple: agent RL should take the step, not the token, as the basic unit. StepPO achieves step-alignment on three levels:

Step-level MDP: an action is an entire step (e.g., a tool call and its result), and state transitions describe how the environment changes after the step—not what the next token is.
Step-level credit assignment: the key innovation. Instead of splitting a reward across the 50 tokens of a step—an almost impossible attribution problem—the whole step shares one reward.
Step-level replay buffer: training data is stored and sampled as complete steps, avoiding the re-tokenization problem where the same step becomes different token sequences in different contexts.

Preliminary Experiments: Notable Gains

Experiments on Agent-R1 and Claw-R1 (built on OpenClaw) show step-level PPO significantly outperforms token-level PPO on agent tasks. Unsurprising: when the objective shifts from "saying every word right" to "getting every step right," the model focuses on step-level decision quality rather than phrasing details.

Why It Matters

StepPO represents a mindset shift from "language model" to "decision model". When an LLM acts as an agent, it is no longer a machine that talks but a machine that makes decisions—and training should reflect that. As systems like OpenClaw and Claude Code grow more capable, StepPO offers a path for RL training to keep pace.

My Take

This is a position paper—its value lies in framing the problem rather than delivering final answers. Agent RL currently carries a hidden assumption: since agents are built on LLMs, LLM RL methods should suffice. StepPO challenges that: an LLM substrate does not mean the optimization granularity must be token-level.

Cars and bicycles both have wheels, but you shouldn't use a bicycle repair manual on a car.

Paper: arxiv.org/abs/2604.18401 Code: github.com/AgentR1/Agent-R1