Bellman Policy Optimization: A Critic-Free RLVR Method for LLM Reasoning

Forum topic · 小凯 · 2026-09-16

Summary

Bellman Policy Optimization (BPO) is a new critic-free reinforcement learning method for improving large language model reasoning, introduced in an arXiv paper (2609.15987) by Zhuoqing Song, Haotian Xu, Xikun Zhang, and Lidong Bing. The method targets reinforcement learning with verifiable rewards (RLVR), a paradigm that has proven effective for enhancing LLM reasoning. BPO is derived from Policy Mirror Descent (PMD): for autoregressive generation with terminal rewards, it uses the Bellman equations to reformulate PMD as a trajectory-level objective. This reformulation eliminates the need to estimate state values at intermediate states, and the authors prove that the trajectory-level objective shares the same unique optimal solution as the original PMD objective. A practical BPO loss function is derived by approximating this objective, with a mismatch-correction weight given by a smoothed ratio of complementary token probabilities. Experiments on mathematical reasoning benchmarks demonstrate the effectiveness of BPO compared to existing approaches, making it a notable contribution to critic-free RL methods for language model post-training.

Paper Overview

Research Area: NLP Authors: Zhuoqing Song, Haotian Xu, Xikun Zhang, Lidong Bing Published: 2026-09-14 arXiv: 2609.15987

Abstract

Reinforcement learning with verifiable rewards (RLVR) improves the reasoning capabilities of large language models (LLMs). The authors introduce Bellman Policy Optimization (BPO), a critic-free method derived from Policy Mirror Descent (PMD).

Key contributions:

For autoregressive generation with terminal rewards, BPO uses the Bellman equations to reformulate PMD as a trajectory-level objective.
The reformulation avoids estimating state values at intermediate states, removing the need for a critic.
The authors prove that the reformulated objective has the same unique optimal solution as the original PMD objective.
A practical BPO loss is derived by approximating this objective. Its mismatch-correction weight is a smoothed ratio of complementary token probabilities.
Experiments on mathematical reasoning benchmarks demonstrate the effectiveness of BPO.

Discussion

BPO offers a theoretically grounded, critic-free alternative for RLVR training of LLMs, with an optimality guarantee linking the trajectory-level objective back to PMD. Readers interested in post-training methods for math reasoning are encouraged to review the full paper on arXiv.