Overview
Field: Machine Learning
Authors: Shiyuan Feng, Huan-ang Gao, Haohan Chi, Hanlin Wu, Zhilong Zhang, Zheng Jiang, Bingxiang He, Wei-Ying Ma, Ya-Qin Zhang, Hao Zhou
Published: 2026-07-06
arXiv: 2607.05394
Summary
Reinforcement learning with verifiable rewards (RLVR) is a powerful method for improving the reasoning ability of language models. However, generating the large volume of rollouts needed to train a new large model is costly and must be repeated whenever a new target model is trained.
This paper proposes a weak-to-strong alternative: first run RL on a smaller teacher model where rollout cost is lower, then transfer the learned knowledge to a larger target student model.
Problem
Directly distilling the post-RL teacher model is not sufficient, because the teacher's final policy conflates the gains introduced by RL with the intrinsic limitations of the small model.
Method: Direct On-Policy Distillation (Direct-OPD)
Direct-OPD transfers the RL-induced policy shift rather than the teacher's final policy. It compares the post-RL teacher with its pre-RL reference version and treats the log-ratio of their policies as a dense implicit reward signal for the student.
Key Results
- Direct-OPD consistently improves the strong student target beyond what the weak teacher alone provides.
- On eight A100 GPUs, in only four hours, Direct-OPD raised Qwen3-1.7B accuracy on AIME 2024 from 48.3% to 62.4%.
Takeaway
Distilling the policy *delta* induced by RL on a small model is a compute-efficient way to uplift larger models on reasoning tasks, avoiding the full cost of running RLVR at the target scale.