Score Centering Stabilizes Off-policy Reinforcement Learning

Forum topic · 小凯 · 2026-09-19

Summary

This arXiv paper (2609.20807) by Martin Marek and Max Ryabinin addresses the training-inference mismatch (TIM) problem in reinforcement learning for large language models. RL training of LLMs is highly sensitive to small numerical differences between training and inference engines, but fully eliminating TIM comes at a major cost to rollout efficiency. The authors show that RL instability under TIM is primarily caused by drift: a persistent bias between training and inference engines that accumulates with every training step. They derive an additive "score centering" correction term that stabilizes RL under TIM by canceling drift. In experiments training models from 0.6B to 30B parameters, score centering alone matches or outperforms importance-sampling-based methods under quantization, with the advantage growing as the mismatch worsens. Because the correction is additive, score centering can also be combined with importance sampling; in staleness experiments, the combination outperforms a pure importance-sampling baseline. The paper offers a practical, low-cost technique for stabilizing off-policy RL training of large language models.

Paper Overview

Field: Machine Learning Authors: Martin Marek, Max Ryabinin Published: 2026-09-17 arXiv: 2609.20807

Abstract

Reinforcement learning (RL) of large language models is notoriously sensitive to small differences between training and inference engines, often referred to as the training-inference mismatch (TIM). However, completely eliminating TIM is impractical, as it would come at a major cost to rollout efficiency. In this paper, we show that the instability of RL under TIM is primarily caused by drift: a persistent bias between training and inference engines that accumulates with every training step. We derive an additive "score centering" correction term that stabilizes RL under TIM by canceling drift. When training models from 0.6B to 30B parameters, score centering alone matches or outperforms methods based on importance sampling under quantization, with the gap growing as the mismatch becomes more severe. Since the correction is additive, score centering can also be combined with importance sampling — in our staleness experiments, this combination outperforms a pure importance-sampling baseline.

Key Takeaways

Root cause identified: RL instability under training-inference mismatch is mainly driven by *drift*, a persistent bias that accumulates across training steps, rather than random per-step noise.
Simple fix: A derived additive "score centering" correction term cancels drift and stabilizes training without requiring the mismatch to be fully eliminated.
Efficiency preserved: Unlike approaches that eliminate TIM entirely, score centering avoids a major hit to rollout efficiency.
Strong results at scale: For models from 0.6B to 30B parameters, score centering alone matches or beats importance-sampling methods under quantization, and the advantage widens with more severe mismatch.
Composable: The additive correction can be combined with importance sampling, outperforming pure importance-sampling baselines in staleness experiments.

--- *Auto-collected on 2026-09-19*