How to Train a Critic Stably and Efficiently: Best-Practice Critic Optimization (BPCO)

Forum topic · 小凯 · 2026-08-26

Summary

This paper (arXiv:2508.17631) by Penghui Qi, Xiangxin Zhou, and Wee Sun Lee addresses the instability of critic-based reinforcement learning for large language models. Group-based methods such as GRPO avoid training a critic by sampling multiple responses per prompt, but a reliable critic could estimate token-level advantages from a single response. The authors analyze why standard critic training is unstable and propose Best-Practice Critic Optimization (BPCO), combining DPPO, value predictions bounded to the reward range, Monte Carlo value targets, unnormalized policy advantages, and length-adaptive generalized advantage estimation. Since the critic is used only during training, BPCO can condition it on reward-defining information, such as reference answers or grading rubrics, hidden from the policy model. On mathematical reasoning tasks with models from 1.5B to 30B-A3B mixture-of-experts, BPCO stably improves critic-based baselines and matches or exceeds group-based baselines while sampling only one response per prompt.

Paper Overview

Field: NLP Authors: Penghui Qi, Xiangxin Zhou, Wee Sun Lee Published: 2025-08-26 arXiv: 2508.17631

Introduction

Group-based reinforcement learning methods such as GRPO for large language models avoid training a critic by sampling multiple responses for each prompt. A reliable critic could instead estimate token-level advantages from one response, but standard critic-based training recipes are often unstable.

Key Contributions

The authors study this instability and develop Best-Practice Critic Optimization (BPCO), a training recipe that combines:

DPPO (distributed/deterministic policy optimization component)
Value predictions bounded to the reward range
Monte Carlo value targets
Unnormalized policy advantages
Length-adaptive generalized advantage estimation (GAE)

Because the critic is used only during training, BPCO can also condition it on reward-defining information — such as a reference answer or grading rubric — that is hidden from the policy model.

Results

Controlled experiments isolate the effect of each design choice. On mathematical reasoning tasks across models ranging from 1.5B to 30B-A3B mixture-of-experts:

BPCO stably improves critic-based baselines
It matches or exceeds group-based baselines while sampling only one response per prompt, improving sampling efficiency