Stable Asynchrony: Variance-Controlled Off-Policy RL for LLMs (VCPO)

Forum topic · 小凯 · 2026-06-24

Summary

VCPO (Variance Controlled Policy Optimization) is a stabilization method for asynchronous reinforcement learning of large language models, proposed by Luke Huang, Zhuoyang Zhang, Qinghao Hu, Shang Yang, and Song Han (arXiv:2602.17616). While asynchronous RL training boosts end-to-end throughput, critic-free policy-gradient methods like REINFORCE and GRPO suffer from sharply higher gradient-estimator variance under high asynchrony: training on stale rollouts produces heavy-tailed importance ratios, allowing a small fraction of samples to dominate updates. VCPO addresses this with two mechanisms: (i) scaling the learning rate based on effective sample size to dampen unreliable updates, and (ii) applying a closed-form minimum-variance baseline designed for the off-policy setting. Across math, general reasoning, and tool-use tasks, VCPO substantially improves the robustness of asynchronous training, cutting long-context multi-turn training time by 2.5x while matching synchronous-training performance.

Paper Overview

Research Area: LLM Authors: Luke Huang, Zhuoyang Zhang, Qinghao Hu, Shang Yang, Song Han Published: 2026-02-19 arXiv: 2602.17616

Abstract

Asynchronous RL training is attractive because it increases end-to-end throughput. However, for critic-free policy-gradient methods such as REINFORCE and GRPO, high asynchrony makes the policy-gradient estimator markedly higher variance: training on stale rollouts creates heavy-tailed importance ratios, causing a small fraction of samples to dominate updates.

The authors propose VCPO (Variance Controlled Policy Optimization), a general stabilization method that:

1. Scales the learning rate based on effective sample size to dampen unreliable updates. 2. Applies a closed-form minimum-variance baseline designed for the off-policy setting.

Results

VCPO substantially improves robustness for asynchronous training across math, general reasoning, and tool-use tasks, reducing long-context, multi-turn training time by 2.5x while matching synchronous performance.

--- *Auto-collected on 2026-06-24*