Beyond GRPO and On-Policy Distillation: An Empirical Sparse-to-Dense Reward Allocation Framework for LLM Training

Forum topic · 小凯 · 2026-05-14

Summary

A forum post summarizes arXiv paper 2605.12483, which proposes a reward-density principle for allocating scarce labeled verifiable training data in LLM training. The authors—Yuanda Xu, Hejian Sang, Zhengze Zhou, Ran He, Zhipeng Wang, and Alborz Geramifard—argue that GRPO-style sparse sequence-level RL and OPD-style dense token-level teacher supervision are not competing recipes but different reward-density regimes. Their allocation rule: spend scarce labels upstream on the strongest model to shape behavior via sparse RL, then transfer that behavior downstream to a smaller deployment student using dense supervision. Experiments with Qwen3 and Llama models on verifiable math show that RL-improving an 8B teacher with bridging distillation outperforms running GRPO directly on a fixed Qwen3-1.7B student. A bridging procedure combining forward-KL warmup on teacher rollouts with OPD on student rollouts performs best on MATH and AIME, and makes subsequent student-side sparse RL effective: GRPO on a cold student, weak before bridging, lifts MATH from 75.4% to 78.5%, beating a matched replay control by 2.8 points.

Beyond GRPO and On-Policy Distillation: An Empirical Sparse-to-Dense Reward Allocation Framework

Field: Machine Learning Authors: Yuanda Xu, Hejian Sang, Zhengze Zhou, Ran He, Zhipeng Wang, Alborz Geramifard Published: 2026-05-12 arXiv: 2605.12483

Key Idea

In settings where labeled verifiable training data is the binding constraint, each checked example should be allocated carefully. The standard practice is to use this data directly on the deployment model (e.g., running GRPO on the deployment student). The authors argue this is often inefficient because it overlooks a reward-density principle:

Sparse sequence-level reward should train models where exploration is productive (i.e., strong upstream models).
Dense token-level teacher reward should be used where the aim is to compress behavior into a smaller model.

Under this view, GRPO-style sparse RL and OPD-style dense teacher supervision are not separate recipes; they are different reward-density regimes. The allocation rule is simple: use scarce labeled training data upstream on the strongest model to shape behavior via sparse RL, then transfer that behavior downstream to a smaller deployment student with dense supervision.

Empirical Findings (Qwen3 and Llama on Verifiable Math)

At a fixed Qwen3-1.7B deployment student scale, RL-improving an 8B teacher followed by dense bridging distillation outperforms running GRPO directly on the student.
Transferring from the same teacher *before* RL underperforms — the bridge works best after upstream RL improvement.
Bridging matters: forward-KL warmup on teacher rollouts plus OPD on student rollouts is consistently strongest on MATH and yields the best pre-Stage-3 AIME endpoints for both 8B and 14B teachers.
Bridging makes subsequent student-side sparse RL effective: GRPO, weak on a cold student, improves MATH from 75.4% to 78.5% after bridging, exceeding a matched replay control by 2.8 points.

Takeaway

Treat sparse RL and on-policy distillation as complementary stages in a sparse-to-dense pipeline rather than alternatives: explore with sparse rewards where it pays off, compress with dense supervision where it counts.

--- *Auto-collected on 2026-05-14*