Neglected Free Lunch from Post-training: Progress Advantage for LLM Agents

论文概要

研究领域: 机器学习
作者: Changdae Oh, Wendi Li, Seongheon Park
发布时间: 2026-06-25
arXiv: 2606.19225

中文摘要

过程奖励模型实现对LLM的细粒度、逐步评估，但为智能体设置构建它们仍然极其困难：长程交互、不可逆动作和随机环境反馈使得人工标注和蒙特卡洛估计在大规模上都不切实际。本工作中，我们表明强化学习（RL）后训练已经提供了有效逐步评分的要素，完全消除了对专用奖励模型训练的需求。具体而言，我们在一般随机马尔可夫决策过程下推导出一个隐式优势，我们称之为进展优势——RL训练策略与其参考策略之间的对数概率比精确恢复了最优优势函数。这种公式化使 resulting 信号无需标注、与领域无关，并作为标准RL后训练流程的副产品可用。我们在三个不同应用上验证了进展优势的有效性：测试时扩展、不确定性量化和五个基准及四个模型家族上的失败归因。在所有设置中，它始终优于基于置信度的基线，且尽管不需要任务特定训练，仍超越了专用训练的奖励模型。我们用对进展优势特征的更深入分析补充这些结果，为在真实世界智能体系统中采用提供实用指导。

原文摘要

Process reward models enable fine-grained, step-level evaluation of LLMs, yet building them for agentic settings remains prohibitively difficult: long-horizon interactions, irreversible actions, and stochastic environment feedback make both human annotation and Monte Carlo estimation infeasible at scale. In this work, we show that reinforcement learning (RL) post-training already provides the ingredients for effective step-level scoring, eliminating the need for dedicated reward model training altogether. Concretely, we derive an implicit advantage under a general stochastic Markov decision process, which we term progress advantage -- log-probability ratio between the RL-trained policy and its reference policy exactly recovers the optimal advantage function. This formulation makes the resu...

自动采集于 2026-06-26

#论文 #arXiv #机器学习 #小凯