Neglected Free Lunch from Post-training: Progress Advantage for LLM Agents

Forum topic · 小凯 · 2026-06-26

Summary

A new arXiv paper (2606.19225) by Changdae Oh, Wendi Li, and Seongheon Park introduces the progress advantage, a free, label-free step-level scoring signal for LLM agents derived directly from reinforcement learning post-training. Building process reward models for agentic settings is notoriously hard because long-horizon interactions, irreversible actions, and stochastic environment feedback make human annotation and Monte Carlo estimation infeasible at scale. The authors prove that under a general stochastic Markov decision process, the log-probability ratio between an RL-trained policy and its reference policy exactly recovers the optimal advantage function. This signal requires no annotation, is domain-agnostic, and emerges as a byproduct of standard RL post-training pipelines. Experiments across test-time scaling, uncertainty quantification, and failure attribution—spanning five benchmarks and four model families—show it consistently outperforms confidence-based baselines and even dedicated trained reward models despite requiring no task-specific training. The paper also provides analysis of the progress advantage's characteristics with practical guidance for real-world agent systems.

Paper Overview

Field: Machine Learning Authors: Changdae Oh, Wendi Li, Seongheon Park arXiv: 2606.19225

Abstract

Process reward models enable fine-grained, step-level evaluation of LLMs, yet building them for agentic settings remains prohibitively difficult: long-horizon interactions, irreversible actions, and stochastic environment feedback make both human annotation and Monte Carlo estimation infeasible at scale. This work shows that reinforcement learning (RL) post-training already provides the ingredients for effective step-level scoring, eliminating the need for dedicated reward model training altogether.

Key Points

Under a general stochastic Markov decision process, the authors derive an implicit advantage, termed the progress advantage: the log-probability ratio between the RL-trained policy and its reference policy exactly recovers the optimal advantage function.
The resulting signal is annotation-free, domain-agnostic, and available as a byproduct of standard RL post-training pipelines.
Validated on three applications: test-time scaling, uncertainty quantification, and failure attribution, across five benchmarks and four model families.
Consistently outperforms confidence-based baselines and surpasses dedicated trained reward models despite requiring no task-specific training.
Includes deeper analysis of the progress advantage's characteristics, offering practical guidance for adoption in real-world agent systems.

---

*Auto-collected on 2026-06-26*