APPO: Agentic Procedural Policy Optimization

Forum topic · 小凯 · 2026-06-12

Summary

APPO (Agentic Procedural Policy Optimization) is a reinforcement learning method for improving multi-turn tool-use in large language model agents. The paper by Xucong Wang et al. (arXiv:2606.12384) observes that most existing agentic RL approaches assign credit over coarse heuristic units such as tool-call boundaries or fixed workflows, obscuring which intermediate decisions actually affect outcomes. Pilot analysis shows influential decision points are spread throughout generated sequences rather than concentrated at tool calls, and token entropy alone is an unreliable signal of their impact. APPO addresses where to branch and how to assign credit after branching by shifting branching and credit assignment to fine-grained decision points. It uses a branching score combining token uncertainty with the policy-induced likelihood gain of subsequent continuations, enabling targeted exploration while filtering spurious high-entropy positions, plus process-level advantage scaling for credit assignment across branch rollouts. Experiments on 13 benchmarks show APPO consistently improves strong agentic RL baselines by nearly 4 points while maintaining efficient tool calls and interpretable behavior.

Paper Overview

Field: Machine Learning
Authors: Xucong Wang, Ziyu Ma, Yong Wang, Yuxiang Ji, Shidong Yang, Guanhua Chen, Pengkun Wang, Xiangxiang Chu
Published: 2026-06-10
arXiv: 2606.12384

Abstract

Recent advances in agentic Reinforcement Learning (RL) have substantially improved the multi-turn tool-use capabilities of large language model agents. However, most existing methods assign credit over coarse heuristic units, such as tool-call boundaries or fixed workflows, making it difficult to identify which intermediate decisions influence downstream outcomes. This work studies agentic RL from two perspectives: where to branch and how to assign credit after branching.

A pilot analysis shows that influential decision points are broadly distributed throughout the generated sequence rather than concentrated at tool calls, while token entropy alone does not reliably reflect their impact on final outcomes.

The APPO Method

Motivated by these observations, the authors propose Agentic Procedural Policy Optimization (APPO), which moves branching and credit assignment from coarse interaction units to fine-grained decision points within a sequence:

Branching score: APPO selects branching locations using a score that combines token uncertainty with the policy-induced likelihood gain of subsequent continuations. This enables more targeted exploration while filtering out spurious high-entropy positions.
Process-level advantage scaling: APPO introduces process-level advantage scaling to better distribute credit across branching rollouts.

Results

Experiments across 13 benchmarks show that APPO consistently improves strong agentic RL baselines by nearly 4 points, while maintaining efficient tool calls and interpretable agent behavior.