Stabilizing Reinforcement Learning with LLMs: Formulation and Practices (Qwen Team)

Forum topic · ✨步子哥 · 2025-12-07

Summary

This post presents a paper from the Qwen Team at Alibaba Inc. proposing a novel formulation for reinforcement learning (RL) with large language models (LLMs). The work explains why and under what conditions the true sequence-level reward can be optimized via a surrogate token-level objective in policy gradient methods such as REINFORCE. The authors introduce a first-order approximation perspective to understand and resolve the instability issues commonly encountered during RL training of LLMs, deriving theoretical conditions under which token-level objectives effectively optimize sequence-level rewards. Experimental results demonstrate significantly improved training stability compared to conventional approaches, with better convergence properties and higher final performance on various benchmark tasks, while maintaining computational efficiency. The paper is authored by Chujie Zheng, Junrong Lin, Kai Dang, Bowen Yu, and colleagues from the Qwen Team. The formulation provides both theoretical foundations and practical guidelines for implementing stable RL training pipelines for LLMs.

Overview

The Qwen Team at Alibaba Inc. (authors: Chujie Zheng, Junrong Lin, Kai Dang, Bowen Yu, Mingze Li, Huiqiang Jiang, Feng Hu, Yuyong Liu, Hao Lin, An Yang, Jingren Zhou, Chen Wu, Junyang Lin) published this work on stabilizing reinforcement learning (RL) with large language models.

Problem

Reinforcement learning has become a key paradigm for enhancing LLMs' ability to tackle complex problem-solving tasks. However, RL training of LLMs often suffers from instability issues, leading to suboptimal performance and training difficulties.

Proposed Formulation

The paper proposes a novel formulation for RL with LLMs, explaining why and under what conditions the true sequence-level reward can be optimized via a surrogate token-level objective in policy gradient methods such as REINFORCE.

Key methodological insight: a first-order approximation perspective is introduced to understand and solve the instability puzzle. The authors derive theoretical conditions under which token-level objectives can effectively optimize sequence-level rewards.

Results

Significantly improved training stability compared to conventional approaches.
Better convergence properties and higher final performance on various benchmark tasks.
The first-order approximation captures the essential dynamics of the RL process while maintaining computational efficiency.

Conclusion

The formulation addresses RL instability from a first-order approximation perspective, validated by both theoretical analysis and experiments. It offers practical guidelines for implementing stable RL training and insights for developing more stable and efficient RL methods for LLMs.

References

[1] OpenAI (2024). Reinforcement Learning from Human Feedback.
[2] Guo et al. (2025). Challenges in RL Training of LLMs.