Rethinking the Divergence Regularization in LLM RL: DRPO (arXiv 2506.04842)

Forum topic · 小凯 · 2026-06-10

Summary

This paper introduces DRPO (Divergence Regularized Policy Optimization), a method for improving trust-region control in reinforcement learning for post-training large language models. LLM RL is typically off-policy due to training-inference mismatch and policy staleness, so stable optimization requires trust-region constraints. Mainstream approaches like PPO and GRPO rely on ratio clipping, but the importance ratio can be a poor proxy for distributional shift in long-tailed vocabularies. Recent work such as DPPO replaces ratio-based clipping with a divergence-based mask that defines the trust region via a token's absolute probability shift. However, DPPO uses a hard mask: once a token crosses the trust-region boundary in a harmful direction, its gradient is discarded rather than corrected. DRPO replaces this hard mask with a smooth, advantage-weighted quadratic regularizer, preserving DPPO's trust-region geometry while producing bounded, continuous gradient weights that damp divergent updates and provide corrective signals outside the boundary. Experiments across model scales, architectures, and precision settings show that DRPO improves both the stability and efficiency of LLM RL training. Paper: arXiv 2506.04842.

Paper Overview

Field: Machine Learning Authors: Jiarui Yao, Xiangxin Zhou, Penghui Qi Published: 2025-06-06 arXiv: 2506.04842

Abstract

Reinforcement learning (RL) has become a key component of post-training large language models (LLMs). In practice, LLM RL is often off-policy because of training-inference mismatch and policy staleness, making trust-region control essential for stable optimization. Mainstream methods such as PPO and GRPO approximate this control with a ratio-clipping mechanism, but the importance ratio can be a poor proxy for distributional shift in long-tailed vocabularies. Recent work such as DPPO addresses this mismatch by replacing ratio-based clipping with a divergence-based mask, yielding a trust region defined by the sampled token's absolute probability shift. However, DPPO still relies on a hard mask: once a token crosses the trust-region boundary in a harmful direction, its gradient is discarded rather than corrected.

Key Contribution: DRPO

The paper proposes Divergence Regularized Policy Optimization (DRPO), which replaces the hard mask with a smooth, advantage-weighted quadratic regularizer. Key properties:

Maintains the same trust-region geometry as DPPO
Produces bounded, continuous gradient weights
Damps divergent updates while providing corrective signals for tokens outside the boundary (instead of discarding their gradients)

Experiments across model scales, architectures, and precision settings demonstrate that DRPO improves the stability and efficiency of LLM RL training.

---

*Auto-collected on 2026-06-10*