[论文] Bellman Policy Optimization

论文概要

研究领域: NLP 作者: Zhuoqing Song, Haotian Xu, Xikun Zhang, Lidong Bing 发布时间: 2026-09-14 arXiv: 2609.15987

中文摘要

基于可验证奖励的强化学习（RLVR）提升了大语言模型的推理能力。我们提出 Bellman 策略优化（BPO），一种源自策略镜像下降（PMD）的无评论家方法。对于具有终端奖励的自回归生成，BPO 利用 Bellman 方程将 PMD 重构为轨迹级目标函数。该重构避免了对中间状态价值函数的估计。我们证明了它与原始 PMD 目标具有相同的唯一最优解。我们通过近似该目标推导出实用的 BPO 损失函数，其失配校正权重是互补令牌概率的平滑比率。在数学推理基准上的实验验证了 BPO 的有效性。

原文摘要

Reinforcement learning with verifiable rewards (RLVR) improves the reasoning capabilities of large language models (LLMs). We introduce Bellman Policy Optimization (BPO), a critic-free method derived from Policy Mirror Descent (PMD). For autoregressive generation with terminal rewards, BPO uses the Bellman equations to reformulate PMD as a trajectory-level objective. The reformulation avoids estimating state values at intermediate states. We prove that it has the same unique optimal solution as the original PMD objective. We derive the practical BPO loss by approximating this objective. Its mismatch-correction weight is a smoothed ratio of complementary token probabilities. Experiments on mathematical reasoning benchmarks demonstrate the effectiveness of BPO.

#论文 #arXiv #NLP #小凯