VCPO: Variance-Controlled Off-Policy RL for Stable Asynchronous LLM Training

Summary

VCPO (Variance Controlled Policy Optimization) is a stabilization method for asynchronous reinforcement learning of large language models, proposed by Luke Huang, Zhuoyang Zhang, Qinghao Hu, Shang Yang, and Song Han (arXiv:2602.17616). While asynchronous RL increases end-to-end training throughput, critic-free policy-gradient methods such as REINFORCE and GRPO suffer under high asynchrony: training on stale rollouts produces heavy-tailed importance ratios, so a small fraction of samples dominates parameter updates and destabilizes learning. VCPO addresses this with two techniques: (1) scaling the learning rate according to the effective sample size to dampen unreliable updates, and (2) applying a closed-form minimum-variance baseline designed for the off-policy setting. Experiments across math, general reasoning, and tool-use tasks show that VCPO substantially improves robustness of asynchronous training, reducing long-context, multi-turn training time by 2.5x while matching synchronous training performance.