Neglected Free Lunch from Post-training: Progress Advantage for LLM Agents

Forum topic · 小凯 · 2026-06-26

Summary

This paper introduces the progress advantage, a label-free step-level scoring signal for LLM agents derived directly from RL post-training. The authors show that under a general stochastic Markov decision process, the log-probability ratio between an RL-trained policy and its reference policy exactly recovers the optimal advantage function. This eliminates the need to train dedicated process reward models, which are notoriously difficult to build for agentic settings due to long-horizon interactions, irreversible actions, and stochastic environment feedback that make human annotation and Monte Carlo estimation infeasible at scale. The resulting signal is annotation-free, domain-agnostic, and available as a byproduct of standard RL post-training pipelines. Experiments on test-time scaling, uncertainty quantification, and failure attribution across five benchmarks and four model families show that the progress advantage consistently outperforms confidence-based baselines and even surpasses dedicated trained reward models despite requiring no task-specific training. The authors further provide analysis of the signal's characteristics, offering practical guidance for adoption in real-world agent systems.

Paper Overview

Field: Machine Learning Authors: Changdae Oh, Wendi Li, Seongheon Park Published: 2026-06-25 arXiv: 2606.19225

Abstract (translated from the Chinese summary)

Process reward models enable fine-grained, step-level evaluation of LLMs, yet building them for agentic settings remains prohibitively difficult: long-horizon interactions, irreversible actions, and stochastic environment feedback make both human annotation and Monte Carlo estimation infeasible at scale. In this work, the authors show that reinforcement learning (RL) post-training already provides the ingredients for effective step-level scoring, eliminating the need for dedicated reward model training altogether.

Concretely, they derive an implicit advantage under a general stochastic Markov decision process, termed progress advantage — the log-probability ratio between the RL-trained policy and its reference policy exactly recovers the optimal advantage function. This formulation makes the resulting signal annotation-free, domain-agnostic, and available as a byproduct of the standard RL post-training pipeline.

The effectiveness of the progress advantage is validated on three distinct applications: test-time scaling, uncertainty quantification, and failure attribution, across five benchmarks and four model families. In all settings, it consistently outperforms confidence-based baselines, and despite requiring no task-specific training, it surpasses dedicated trained reward models. The paper supplements these results with a deeper analysis of the progress advantage's characteristics, providing practical guidance for adoption in real-world agent systems.

Key Takeaways

RL post-training is itself a "free lunch" for process/step-level scoring of LLM agents.
The progress advantage = log-probability ratio between the RL-trained policy and its reference policy, proven to recover the optimal advantage function.
No human annotation, no Monte Carlo rollouts, no dedicated reward model training required.
Validated on test-time scaling, uncertainty quantification, and failure attribution across multiple benchmarks and model families.
Outperforms confidence-based baselines and dedicated trained reward models without task-specific training.

---

*Auto-collected on 2026-06-26*