Bebop: Accelerating RL Training via Multi-Token Prediction with Rejection Sampling

Forum topic · 小凯 · 2026-06-12

Summary

Reinforcement learning (RL) is central to modern large language models, but the rollout stage remains the key bottleneck in RL training. Although Multi-Token Prediction (MTP) can accelerate rollouts via speculative decoding, its acceptance rates often degrade significantly during RL training, limiting speedup. This paper presents Bebop, a systematic study of MTP in LLM post-training with practical recipes for integrating MTP into large-scale RL pipelines. The authors show that MTP acceptance rate is fundamentally bounded by model entropy fluctuation, exhibiting a clear negative linear relationship with entropy rise during RL. They demonstrate that probabilistic rejection sampling largely mitigates entropy-induced disturbances compared to greedy draft sampling, and propose an end-to-end TV loss that directly optimizes multi-step rejection-sampling acceptance, yielding roughly 10% acceptance improvement, up to 95% acceptance rates, and up to 25% additional inference throughput on math reasoning, code generation, and agentic tasks. Pre-RL MTP training with TV loss and rejection sampling maintains consistent acceptance throughout RL, eliminating costly online MTP updates. Experiments show up to 1.8x end-to-end speedup in asynchronous RL training on Qwen models.

Overview

Field: Machine Learning
Authors: Yucheng Li, Huiqiang Jiang, Yang Xu, Jianxin Yang, Yi Zhang, Yizhong Cao, Yuhao Shen, Fan Zhou, Rui Men, Jianwei Zhang, An Yang, Bowen Yu, Bo Zheng, Fei Huang, Junyang Lin, Dayiheng Liu, Jingren Zhou
Published: 2026-06-10
arXiv: 2606.12370

Summary

Reinforcement learning (RL) has become a key component in modern large language models, yet the rollout stage remains the key bottleneck in RL training pipelines. Although Multi-Token Prediction (MTP) offers a natural solution to accelerate rollouts through speculative decoding, many studies have observed that MTP acceptance rates degrade significantly during RL training, leading to limited speedup.

To address this bottleneck, the authors present Bebop, a systematic study of MTP in LLM post-training, and offer practical recipes to integrate MTP into large-scale RL pipelines:

1. Entropy bounds acceptance: The MTP acceptance rate is fundamentally bounded by model entropy fluctuation, showing a clear negative linear relationship with the rise of entropy during the RL stage. 2. Rejection sampling over greedy decoding: Probabilistic rejection sampling largely mitigates the disturbances introduced by entropy in RL, compared to greedy draft sampling. 3. End-to-end TV loss: Traditional MTP training objectives (cross-entropy or KL) are suboptimal in this setting. A novel end-to-end TV loss directly optimizes multi-step rejection-sampling acceptance, yielding ~10% acceptance rate improvement, up to 95% acceptance rates, and up to 25% additional inference throughput gains in math reasoning, code generation, and agentic tasks. 4. Pre-RL MTP training suffices: Among various online MTP training strategies tested during RL, pre-RL MTP training with end-to-end TV loss and rejection sampling achieves consistent acceptance rates and speedup throughout RL, eliminating the need for costly online MTP updates.

Experiments demonstrate up to 1.8x end-to-end speedup in asynchronous RL training on Qwen3.5, Qwen3.6, and Qwen3.7 models.

*Auto-collected on 2026-06-12.*

Bebop: Accelerating RL Training via Multi-Token Prediction with Rejection Sampling

Summary

Overview

Summary

Tags