Mu-GRPO: How Off-Policy Can GRPO Get? 2x Faster LLM RL Training with Stale Data

Forum topic · 小凯 · 2026-05-19

Summary

Mu-GRPO is a variant of GRPO, the reinforcement learning method behind DeepSeek-R1's reasoning capabilities, that deliberately uses stale (off-policy) rollout data to cut training time roughly in half. Standard GRPO is near-on-policy: after every policy update, fresh samples must be generated, causing costly switching between rollout generation and optimization. Mu-GRPO restructures training into a small number (e.g., four) of large sequential generate-then-optimize phases: massive rollouts are produced once per phase, then optimized over many consecutive steps. Two stabilization techniques are introduced: relaxed clipping, which widens PPO's probability-ratio clipping range so useful gradients from stale rollouts survive, and negative-advantage veto, which suppresses suffix updates that attempts to cancel out negative advantage signals. Across five language models and multiple math reasoning benchmarks, Mu-GRPO matches or exceeds standard GRPO performance while achieving about 2x wall-clock speedup. Open questions remain: optimal number of phases, automatic tuning of the relaxed clipping parameter, and whether the veto mechanism generalizes beyond mathematical reasoning tasks.

Background: GRPO and its on-policy bottleneck

GRPO is the reinforcement learning training method behind DeepSeek-R1's reasoning capability. It samples N responses per prompt, groups them, and uses group-normalized rewards for advantage estimation.

However, GRPO has an efficiency problem: it is essentially a near-on-policy method. After every policy update, new data must be sampled, otherwise the advantage estimates from old data become inaccurate. This forces frequent switching between sampling and optimization, incurring large system overhead.

The counterintuitive idea

Tian, Xie, and Wei asked: what if we deliberately use stale data?

Mu-GRPO organizes training into a small number of large sequential generation-optimization phases (e.g., 4). At the start of each phase, a large batch of rollouts is generated once, then optimized over multiple consecutive steps, after which the updated policy performs the next large generation. Between phases the policy has changed substantially — rollout staleness is high — but the generation-optimization switching overhead is drastically reduced.

Two stabilization techniques

Relaxed clipping — Standard PPO applies ϵ-clipping to the probability ratio to prevent overly large updates, but stale rollouts need more room to change to learn effectively. Relaxed clipping preserves useful stale gradients.
Negative-advantage veto — It is observed that in the second half of negative-advantage responses, the model tends to generate suffix updates attempting to cancel the advantage signal. Mu-GRPO directly vetoes these suffix updates.

Results

Across 5 language models and multiple math reasoning benchmarks, Mu-GRPO matches or exceeds standard GRPO's performance while achieving approximately 2x wall-clock training speedup.

Open questions

Is "4 phases" optimal? More phases approach on-policy behavior (higher fidelity but higher overhead); where is the turning point where fewer phases become unstable?
How should the relaxed clipping parameter be set? Too loose and training diverges; too tight and stale data is useless. Can it be determined automatically?
Does the veto mechanism apply to coding or general RL tasks, or is it specific to mathematical reasoning?

References

1. Tian, M., Xie, Y., & Wei, C. (2026). *How Off-Policy Can GRPO Be? Mu-GRPO for Efficient LLM Reinforcement Learning*. arXiv:2605.17570 [cs.LG]. 2. Shao, Z., et al. (2025). *DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning*. arXiv. 3. Schulman, J., et al. (2017). *Proximal Policy Optimization Algorithms*. arXiv.