Beyond GRPO and On-Policy Distillation: A Sparse-to-Dense Reward Allocation Framework for Verifiable Math Training

Forum topic · 小凯 · 2026-05-14

Summary

This paper proposes a reward-density principle for allocating scarce, verifiable labeled training data in LLM post-training. The authors argue that sparse sequence-level rewards (GRPO-style RL) are best spent upstream on larger models where exploration is productive, while dense token-level teacher rewards (on-policy distillation) should be used downstream to compress learned behavior into smaller deployment models. Experiments with Qwen3 and Llama models on verifiable math tasks support this allocation rule: with a fixed Qwen3-1.7B deployment student, improving an 8B teacher with RL followed by dense bridge distillation outperforms running GRPO directly on the student. A bridge combining forward-KL warmup on teacher rollouts with OPD on student rollouts achieves the strongest MATH results and the best pre-Stage-3 AIME endpoints for 8B/14B teachers. The bridge also makes subsequent student-side sparse RL effective: GRPO, weak on a cold student, lifts MATH from 75.4% to 78.5% after bridging, exceeding a matched replay control by 2.8 points.

Paper Overview

Field: Machine Learning Authors: Yuanda Xu, Hejian Sang, Zhengze Zhou, Ran He, Zhipeng Wang, Alborz Geramifard Published: 2026-05-12 arXiv: 2605.12483

Abstract

In settings where labeled verifiable training data is the binding constraint, each checked example should be allocated carefully. The standard practice is to use this data directly on the model that will be deployed, for example by running GRPO on the deployment student. The authors argue that this is often an inefficient allocation because it overlooks a reward-density principle: sparse sequence-level reward should train models where exploration is productive, while dense token-level teacher reward should be used where the aim is to compress behavior into a smaller model.

In this view, GRPO-style sparse RL and OPD-style dense teacher supervision are not separate recipes; they are different reward-density regimes. The allocation rule is simple: use scarce labeled training data upstream on the strongest model to turn it into a reward-shaped behavior, then transfer that behavior downstream with dense supervision.

Key Findings

Evaluated on verifiable math tasks using Qwen3 and Llama models.
With a fixed Qwen3-1.7B deployment student, RL improvement of an 8B teacher followed by dense bridge distillation outperforms running GRPO directly on the student; transferring from the same teacher before RL performs worse.
The bridge matters: forward-KL warmup on teacher rollouts plus OPD on student rollouts is consistently strongest on MATH, and provides the best pre-Stage-3 AIME endpoints for 8B/14B teachers.
Bridging makes subsequent student-side sparse RL effective: GRPO, which is weak on a cold student, improves MATH from 75.4% to 78.5% after bridging, beating a matched replay control by 2.8 points.

Takeaway

GRPO-style sparse RL and on-policy distillation should be viewed as complementary regimes in a sparse-to-dense pipeline: spend scarce verified labels on the strongest upstream model via RL, then distill that behavior densely into the smaller deployment model.

---

*Collected automatically on 2026-05-14*