[论文] Score Centering Stabilizes Off-policy Reinforcement Learning

论文概要

研究领域: ML 作者: Martin Marek, Max Ryabinin 发布时间: 2026-09-17 arXiv: 2609.20807

中文摘要

大语言模型的强化学习（RL）对训练引擎和推理引擎之间的微小差异非常敏感，这通常被称为"训练-推理失配"（TIM）。然而，完全消除 TIM 是不切实际的，因为这会以大幅降低 rollout 效率为代价。本文表明，TIM 下 RL 的不稳定性主要由漂移（drift）引起：训练引擎和推理引擎之间的持续性偏差，随每个训练步骤累积。我们推导出一个加性的"分数中心化"（score centering）校正项，通过抵消漂移来稳定 TIM 下的 RL。在训练 0.6B 到 30B 参数的模型时，仅分数中心化就能匹配或优于基于重要性采样的量化方法，且失配越严重，优势差距越大。由于校正是加性的，分数中心化还可以与重要性采样组合使用——在我们的陈旧性（staleness）实验中，它们的组合优于纯重要性采样基线。

原文摘要

Reinforcement learning (RL) of large language models is notoriously sensitive to small differences between training and inference engines, often referred to as the training-inference mismatch (TIM). However, completely eliminating TIM is impractical, as it would come at a major cost to rollout efficiency. In this paper, we show that the instability of RL under TIM is primarily caused by drift: a persistent bias between training and inference engines that accumulates with every training step. We derive an additive "score centering" correction term that stabilizes RL under TIM by canceling drift. When training models from 0.6B to 30B parameters, score centering alone matches or outperforms methods based on importance sampling under quantization, with the gap growing as the mismatch becomes m...

#论文 #arXiv #ML #小凯