Rethinking Divergence Regularization in LLM RL: DRPO (arXiv 2506.04842)

Forum topic · 小凯 · 2026-06-10

Summary

This paper (arXiv:2506.04842) by Jiarui Yao, Xiangxin Zhou, and Penghui Qi proposes Divergence-regularized Policy Optimization (DRPO), a new method for stabilizing reinforcement learning post-training of large language models. LLM RL is typically off-policy due to training-inference mismatch and policy staleness, making trust-region control essential. Mainstream methods like PPO and GRPO use ratio clipping, but importance ratios can poorly proxy distributional shift in long-tailed vocabularies. DPPO instead uses a divergence-based mask defining trust regions via absolute token probability shifts, yet it discards gradients of tokens crossing the boundary via hard masks. DRPO replaces this hard mask with a smooth advantage-weighted quadratic regularizer that preserves DPPO's trust-region geometry while producing bounded, continuous gradient weights—attenuating divergent updates and providing corrective signals outside the boundary. Experiments across model scales, architectures, and precision settings show improved stability and efficiency of LLM RL training.

Paper Overview

Field: Machine Learning Authors: Jiarui Yao, Xiangxin Zhou, Penghui Qi Published: 2025-06-06 arXiv: 2506.04842

Abstract

Reinforcement learning (RL) has become a key component of post-training large language models (LLMs). In practice, LLM RL is often off-policy because of training-inference mismatch and policy staleness, making trust-region control essential for stable optimization. Mainstream methods such as PPO and GRPO approximate this control with a ratio-clipping mechanism, but the importance ratio can be a poor proxy for distributional shift in long-tailed vocabularies. Recent work such as DPPO addresses this mismatch by replacing ratio-based clipping with a divergence-based mask, yielding a trust region defined by the sampled token's absolute probability shift. However, DPPO still relies on a hard mask: once a token crosses the trust-region boundary in a harmful direction, its gradient is discarded rather than corrected.

To address this, the authors propose Divergence-regularized Policy Optimization (DRPO), which replaces the hard mask with a smooth advantage-weighted quadratic regularizer. DRPO maintains the same trust-region geometry as DPPO while producing bounded, continuous gradient weights that attenuate divergent updates and provide corrective signals outside the boundary. Experiments across model scales, architectures, and precision settings demonstrate that DRPO improves the stability and efficiency of LLM RL training.

Key Contributions

Identifies limitations of ratio clipping (PPO/GRPO) for long-tailed vocabulary distribution shifts
Improves on DPPO's divergence-based hard masking with a smooth quadratic regularizer
Preserves trust-region geometry while enabling gradient correction beyond the boundary
Demonstrates improved training stability and efficiency across diverse model configurations

---

*Auto-collected on 2026-06-10*