Summary
Bellman Policy Optimization (BPO) is a critic-free reinforcement learning method for improving the reasoning capabilities of large language models trained with verifiable rewards (RLVR). Derived from Policy Mirror Descent (PMD), BPO uses the Bellman equations to reformulate PMD as a trajectory-level objective for autoregressive generation with terminal rewards. This reformulation avoids estimating state values at intermediate states, and the authors prove it shares the same unique optimal solution as the original PMD objective. A practical BPO loss is derived by approximating this objective, with a mismatch-correction weight given by a smoothed ratio of complementary token probabilities. Experiments on mathematical reasoning benchmarks demonstrate the effectiveness of BPO. The paper (arXiv:2609.15987) is authored by Zhuoqing Song, Haotian Xu, Xikun Zhang, and Lidong Bing in the NLP domain.
Paper Overview
Research Area: NLP
Authors: Zhuoqing Song, Haotian Xu, Xikun Zhang, Lidong Bing
Published: 2026-09-14
arXiv: 2609.15987
Abstract
Reinforcement learning with verifiable rewards (RLVR) improves the reasoning capabilities of large language models (LLMs). We introduce Bellman Policy Optimization (BPO), a critic-free method derived from Policy Mirror Descent (PMD). For autoregressive generation with terminal rewards, BPO uses the Bellman equations to reformulate PMD as a trajectory-level objective. The reformulation avoids estimating state values at intermediate states. We prove that it has the same unique optimal solution as the original PMD objective. We derive the practical BPO loss by approximating this objective. Its mismatch-correction weight is a smoothed ratio of complementary token probabilities. Experiments on mathematical reasoning benchmarks demonstrate the effectiveness of BPO.
---
*Auto-collected on 2026-09-16*
This page is an English static mirror generated for search and AI citation.
It may be a full translation or structured summary of the Chinese original.
Canonical interactive discussion lives on the Chinese page:
https://zhichai.net/topic/178634861