Reinforcement Learning Stability for Large Language Models: Formulation and Practice

Forum topic · ✨步子哥 · 2025-12-07

Summary

This post from the Qwen Team at Alibaba presents a novel formulation for reinforcement learning (RL) in large language models (LLMs), explaining why and under what conditions true sequence-level rewards can be optimized through surrogate token-level objectives in policy gradient methods such as REINFORCE. RL has become a key paradigm for improving LLMs' complex problem-solving abilities, but RL training is frequently unstable, leading to suboptimal performance and difficult optimization. The proposed approach adopts a first-order approximation perspective to analyze and mitigate these instabilities, deriving theoretical conditions under which token-level objectives effectively optimize sequence-level rewards. Experimental results show that the method significantly improves training stability compared to conventional approaches, achieving better convergence and higher final performance across benchmark tasks while maintaining computational efficiency. The authors state that this work provides both a theoretical foundation and practical guidance for implementing stable RL training in LLMs, offering insights for developing more stable and efficient RL training methods. The post is attributed to Chujie Zheng, Junrong Lin, Kai Dang, and other members of the Qwen Team at Alibaba Inc. (2025).

Reinforcement Learning Stability for Large Language Models: Formulation and Practice

Authors: Chujie Zheng, Junrong Lin, Kai Dang, Bowen Yu, Mingze Li, Huiqiang Jiang, Feng Hu, Yuyong Liu, Hao Lin, An Yang, Jingren Zhou, Chen Wu, Junyang Lin — Qwen Team, Alibaba Inc.

Abstract

This work proposes a novel formulation for reinforcement learning (RL) of large language models (LLMs), explaining why and under what conditions true sequence-level rewards can be optimized through surrogate token-level objectives in policy gradient methods (e.g., REINFORCE). The formulation addresses common instability problems in RL training of LLMs, providing both a theoretical foundation and a practical implementation scheme.

Introduction

Reinforcement learning has become a key technical paradigm for enhancing LLMs' ability to solve complex problems. However, RL training of LLMs often suffers from instability, leading to suboptimal performance and difficult training. This paper tackles these challenges by proposing a novel formulation that stabilizes the RL process.

Methodology

The method introduces a first-order approximation perspective for understanding and resolving instability in LLM RL training. The authors derive theoretical conditions under which token-level objectives can effectively optimize sequence-level rewards. This formulation offers practical guidance for implementing stable RL training in LLMs.

Experimental Results

Compared with conventional methods, the proposed approach significantly improves training stability, achieving better convergence and higher final performance across various benchmark tasks. A key finding: the first-order approximation effectively captures the essential dynamics of the RL process while remaining computationally efficient.

Conclusion

The paper presents a novel formulation that stabilizes LLM RL from a first-order approximation perspective. Both theoretical analysis and experimental results validate its effectiveness, providing valuable insights for developing more stable and efficient RL training methods for LLMs.

References

1. OpenAI (2024). Reinforcement Learning from Human Feedback. 2. Guo et al. (2025). Challenges of RL Training for LLMs.

Reinforcement Learning Stability for Large Language Models: Formulation and Practice

Summary

Reinforcement Learning Stability for Large Language Models: Formulation and Practice

Abstract

Introduction

Methodology

Experimental Results

Conclusion

References

Tags