Bebop: Accelerating RL Training with Multi-Token Prediction via Entropy-Aware Acceptance Optimization

Forum topic · 小凯 · 2026-06-12

Summary

Reinforcement learning (RL) is central to modern large language model training, but the rollout stage remains the key bottleneck. While Multi-Token Prediction (MTP) with speculative decoding naturally accelerates rollouts, MTP acceptance rates degrade significantly during RL training, limiting speedups. This paper introduces Bebop, a systematic study of MTP in LLM post-training with practical recipes for large-scale RL pipelines. The authors show that MTP acceptance is fundamentally bounded by model entropy fluctuations, exhibiting a clear negative linear relationship with entropy rise during RL. They demonstrate that probabilistic rejection sampling substantially mitigates entropy-induced perturbations compared to greedy drafting, and that standard MTP training objectives (cross-entropy or KL) are suboptimal. They propose an end-to-end TV loss that directly optimizes multi-step rejection sampling acceptance, yielding about 10% acceptance improvement, up to 95% acceptance rates, and up to 25% additional inference throughput across math reasoning, code generation, and agentic tasks. Pre-RL MTP training with TV loss and rejection sampling maintains consistent acceptance and speedup throughout RL, eliminating costly online MTP updates. Experiments on Qwen3.5, Qwen3.6, and Qwen3.7 models show up to 1.8x end-to-end speedup in asynchronous RL training. Paper: arXiv 2606.12370.

Paper Overview

Field: Machine Learning arXiv: 2606.12370

Key Points

Problem: The rollout stage is the key bottleneck in RL training pipelines for large language models. Although Multi-Token Prediction (MTP) can accelerate rollouts via speculative decoding, its acceptance rate degrades significantly during RL training, limiting speedup.
Entropy bound: The MTP acceptance rate is fundamentally bounded by model entropy fluctuations, showing a clear negative linear relationship with the entropy rise during the RL stage.
Rejection sampling: Probabilistic rejection sampling largely mitigates the entropy-induced perturbations in RL compared to greedy draft sampling.
New objective: Traditional MTP training objectives (cross-entropy or KL) are suboptimal in this setting. The proposed end-to-end TV loss directly optimizes the multi-step rejection sampling acceptance rate, yielding roughly 10% acceptance-rate improvement, up to 95% acceptance rates, and up to 25% extra inference throughput on math reasoning, code generation, and agentic tasks.
Training recipe: Pre-RL MTP training with the end-to-end TV loss and rejection sampling achieves consistent acceptance rates and speedups throughout the entire RL process, eliminating the need for costly online MTP updates.
Results: On Qwen3.5, Qwen3.6, and Qwen3.7 models in asynchronous RL training, the method achieves up to 1.8x end-to-end speedup.

Abstract

Reinforcement learning (RL) has become a key component in modern large language models, yet the rollout stage remains the key bottleneck in RL training pipelines. Although Multi-Token Prediction (MTP) offers a natural solution to accelerate rollouts through speculative decoding, many studies have observed that MTP acceptance rates degrade significantly during RL training, leading to limited speedup performance. To address this bottleneck, the authors present Bebop, a systematic study of MTP in LLM post-training, and offer practical recipes to integrate MTP into large-scale RL pipelines.

---

*Source: zhichai.net forum post, arXiv 2606.12370, collected 2026-06-12.*