Paper Overview
Field: NLP Authors: Penghui Qi, Xiangxin Zhou, Wee Sun Lee Published: 2025-08-26 arXiv: 2508.17631
Summary
Group-based reinforcement learning methods such as GRPO for large language models avoid training a critic by sampling multiple responses for each prompt. A reliable critic could instead estimate token-level advantages from one response, but standard critic-based training recipes are often unstable.
The paper studies this instability and develops Best-Practice Critic Optimization (BPCO), a recipe that combines:
- DPPO
- Value predictions bounded to the reward range
- Monte Carlo value targets
- Unnormalized policy advantages
- Length-adaptive generalized advantage estimation
- Critic-based RL can be made stable with the right combination of optimization practices, eliminating the need to sample multiple responses per prompt.
- Conditioning the critic on privileged reward-defining information (reference answers, rubrics) is possible since the critic is discarded after training.
- Single-response sampling reduces inference cost compared to group-based methods like GRPO.
Because the critic is used only during training, BPCO can also condition it on reward-defining information, such as a reference answer or grading rubric, that is hidden from the policy.
In controlled experiments on math reasoning tasks, with models ranging from 1.5B to 30B-A3B mixture-of-experts, BPCO stably improves critic-based baselines and matches or exceeds group-based baselines while sampling only one response per prompt.