[论文] How to Train a Critic Stably and Efficiently

论文概要

研究领域: NLP 作者: Penghui Qi, Xiangxin Zhou, Wee Sun Lee 发布时间: 2025-08-26 arXiv: 2508.17631

中文摘要

基于群体的强化学习方法（如GRPO）通过为每个提示采样多个回复来避免训练critic。然而，一个可靠的critic本可以从单次回复中估计token级别的优势，但标准的critic训练方法往往不稳定。本文研究了这一不稳定性问题，并提出了Best-Practice Critic Optimization (BPCO)方法。BPCO结合了DPPO、限制在奖励范围内的价值预测、蒙特卡洛价值目标、非归一化策略优势以及长度自适应广义优势估计。由于critic仅在训练时使用，BPCO还可以让它基于奖励定义信息（如参考答案或评分标准）进行条件判断，而这些信息对策略模型是隐藏的。在从1.5B到30B-A3B混合专家模型的数学推理任务上，BPCO稳定提升了critic-based基线，并在每次提示仅采样一个回复的情况下达到或超过了group-based基线。

原文摘要

Group-based reinforcement learning methods such as GRPO for large language models avoid training a critic by sampling multiple responses for each prompt. A reliable critic could instead estimate token-level advantages from one response, but standard critic-based training recipes are often unstable. We study this instability and develop Best-Practice Critic Optimization (BPCO), a recipe that combines DPPO, value predictions bounded to the reward range, Monte Carlo value targets, unnormalized policy advantages, and length-adaptive generalized advantage estimation. Because the critic is used only during training, BPCO can also condition it on reward-defining information, such as a reference answer or grading rubric, that is hidden from the policy. Controlled experiments isolate the effect of ...

--- *自动采集于 2026-08-26*

#论文 #arXiv #NLP #小凯