RL Stability for Large Language Models: Formulation and Practice (Qwen Team)

Forum topic · ✨步子哥 · 2025-12-07

Summary

The Qwen Team at Alibaba presents a novel formulation for reinforcement learning (RL) in large language models (LLMs), addressing common training instability issues. The work explains why and under what conditions true sequence-level rewards can be optimized through surrogate token-level objectives within policy gradient methods such as REINFORCE. Using a first-order approximation perspective, the authors derive theoretical conditions under which token-level objectives effectively optimize sequence-level rewards, providing practical guidance for implementing stable RL training in LLMs. Experimental results show that the proposed approach significantly improves training stability compared to conventional methods, achieving better convergence and higher final performance across various benchmark tasks, while maintaining computational efficiency. The paper argues that the first-order approximation captures the essential dynamics of the RL process. Authored by Chujie Zheng, Junrong Lin, Kai Dang, and other members of the Qwen Team, this work offers both theoretical foundations and practical recipes for more stable and efficient RL-based post-training of large language models.

RL Stability for Large Language Models: Formulation and Practice

Authors: Chujie Zheng, Junrong Lin, Kai Dang, Bowen Yu, Mingze Li, Huiqiang Jiang, Feng Hu, Yuyong Liu, Hao Lin, An Yang, Jingren Zhou, Chen Wu, Junyang Lin — Qwen Team, Alibaba Inc.

Abstract

This work proposes a novel formulation for reinforcement learning (RL) in large language models (LLMs). It explains why and under what conditions true sequence-level rewards can be optimized via surrogate token-level objectives within policy gradient methods such as REINFORCE. The approach addresses the instability issues commonly seen in RL training of LLMs, providing both a theoretical foundation and a practical implementation scheme.

Introduction

RL has become a key paradigm for enhancing the ability of LLMs to solve complex problems. However, RL training of LLMs often suffers from instability, leading to suboptimal performance and training difficulties. This paper tackles these challenges by proposing a novel formulation that stabilizes the RL process.

Methodology

The method introduces a first-order approximation perspective to understand and resolve training instability in LLM RL. The authors derive theoretical conditions under which token-level objectives can effectively optimize sequence-level rewards.

> The formulation provides practical guidance for implementing stable RL training in large language models.

Experimental Results

Experiments demonstrate that the proposed method significantly improves training stability compared to conventional approaches, achieving better convergence behavior and higher final performance across various benchmark tasks.

> The first-order approximation effectively captures the essential dynamics of the RL process while maintaining computational efficiency.

Conclusion

The paper presents a novel formulation for stabilizing RL in LLMs from a first-order approximation perspective. Both theoretical analysis and experimental results validate the effectiveness of the approach, offering valuable insights for developing more stable and efficient RL training methods for large language models.

References

1. OpenAI (2024). Reinforcement Learning from Human Feedback. 2. Guo et al. (2025). Challenges in RL Training of Large Language Models.