[论文] Breaking Entropy Bounds: Accelerating RL Training via MTP with Re...

论文概要

研究领域: ML 作者: Yucheng Li, Huiqiang Jiang, Yang Xu, Jianxin Yang, Yi Zhang, Yizhong Cao, Yuhao Shen, Fan Zhou, Rui Men, Jianwei Zhang, An Yang, Bowen Yu, Bo Zheng, Fei Huang, Junyang Lin, Dayiheng Liu, Jingren Zhou 发布时间: 2026-06-10 arXiv: 2606.12370

中文摘要

强化学习（RL）已成为现代大型语言模型的关键组件，但rollout阶段仍是RL训练流程的关键瓶颈。虽然多token预测（MTP）通过推测解码为加速rollout提供自然解决方案，但许多研究观察到MTP接受率在RL训练期间显著下降，导致有限的速度提升。为解决这一瓶颈，我们呈现Bebop，MTP在LLM后训练中的系统研究，并提供将MTP集成到大规模RL流程的实用方法。首先，我们揭示MTP接受率根本上受模型熵波动限制，这显示与RL阶段熵上升存在清晰负线性关系。其次，我们证明概率拒绝采样相比贪婪草稿采样在很大程度上缓解RL中熵引入的扰动。我们进一步识别传统MTP训练目标（交叉熵或KL）在此类设置中是次优的，因此我们提出新颖端到端TV损失，直接优化多步拒绝采样接受率，产生约10%接受率提升，在数学推理、代码生成和智能体任务中达到高达95%接受率和高达25%额外推理吞吐量增益。第三，我们测试RL期间各种在线MTP训练策略，并表明具有端到端TV损失和拒绝采样的预RL MTP训练在整个RL中实现一致接受率和加速，消除昂贵在线MTP更新的需求。我们提供广泛实验和分析验证我们的发现。实验结果显示我们的方法在Qwen3.5、Qwen3.6和Qwen3.7模型的异步RL训练中实现高达1.8倍端到端加速。

原文摘要

Reinforcement learning (RL) has become a key component in modern large language models, yet the rollout stage remains the key bottleneck in RL training pipelines. Although Multi-Token Prediction (MTP) offers a natural solution to accelerate rollouts through speculative decoding, many studies have observed that MTP acceptance rates degrade significantly during RL training, leading to limited speedup performance. To address this bottleneck, we present Bebop, a systematic study of MTP in LLM post-training, and offer practical recipes to integrate MTP into large-scale RL pipelines. First, we reveal that the MTP acceptance rate is fundamentally bounded by the fluctuation of model entropy, which demonstrates a clear negative linear relationship with the rise of entropy in the RL stage. Second, w...

--- *自动采集于 2026-06-12*

#论文 #arXiv #ML #小凯