Paper Overview
Field: Machine Learning Authors: Martin Marek, Max Ryabinin Published: 2026-09-17 arXiv: 2609.20807
Abstract
Reinforcement learning (RL) of large language models is notoriously sensitive to small differences between training and inference engines, often referred to as the training-inference mismatch (TIM). However, completely eliminating TIM is impractical, as it would come at a major cost to rollout efficiency. In this paper, we show that the instability of RL under TIM is primarily caused by drift: a persistent bias between training and inference engines that accumulates with every training step. We derive an additive "score centering" correction term that stabilizes RL under TIM by canceling drift. When training models from 0.6B to 30B parameters, score centering alone matches or outperforms methods based on importance sampling under quantization, with the gap growing as the mismatch becomes more severe. Since the correction is additive, score centering can also be combined with importance sampling — in our staleness experiments, their combination outperforms pure importance sampling baselines.
Key Contributions
- Identifies drift (a persistent, accumulating bias between training and inference engines) as the primary cause of RL instability under training-inference mismatch (TIM).
- Derives an additive score centering correction term that cancels drift and stabilizes off-policy RL.
- Demonstrates at scale (0.6B–30B parameter models) that score centering alone matches or beats importance sampling based on quantization, with the advantage growing as mismatch increases.
- Shows that score centering composes with importance sampling, outperforming pure importance sampling baselines in staleness experiments.