Stabilizing Reinforcement Learning with LLMs: Formulation and Practices
Authors: Chujie Zheng, Junrong Lin, Kai Dang, Bowen Yu, Mingze Li, Huiqiang Jiang, Feng Hu, Yuyong Liu, Hao Lin, An Yang, Jingren Zhou, Chen Wu, Junyang Lin — Qwen Team, Alibaba Inc.
Abstract
This paper proposes a novel formulation for reinforcement learning (RL) with large language models, explaining why and under what conditions the true sequence-level reward can be optimized via a surrogate token-level objective in policy gradient methods such as REINFORCE. The approach addresses the instability issues commonly encountered in RL training of LLMs, providing both theoretical foundations and practical implementations.
Introduction
Reinforcement learning has become a key technical paradigm for enhancing large language models' (LLMs) ability to tackle complex problem-solving tasks. However, RL training of LLMs often suffers from instability, leading to suboptimal performance and training difficulties. The paper addresses these challenges by proposing a formulation that stabilizes the RL process.
Methodology
The approach introduces a first-order approximation perspective to understand and solve the instability puzzle in RL training of LLMs. The authors derive theoretical conditions under which token-level objectives can effectively optimize sequence-level rewards. The formulation provides practical guidelines for implementing stable RL training.
Results
- The proposed method significantly improves training stability compared to conventional approaches.
- It achieves better convergence properties and higher final performance on various benchmark tasks.
- The first-order approximation captures the essential dynamics of the RL process while maintaining computational efficiency.
Conclusion
The paper presents a novel formulation for stabilizing RL with LLMs from a first-order approximation perspective. Both theoretical analysis and experiments validate its effectiveness, offering insights for developing more stable and efficient RL training methods for LLMs.
References
[1] OpenAI (2024). Reinforcement Learning from Human Feedback. [2] Guo et al. (2025). Challenges in RL Training of LLMs.