Weak-to-Strong Generalization via Direct On-Policy Distillation (Direct-OPD)

Paper Overview

• Field: Machine Learning
• Authors: Shiyuan Feng, Huan-ang Gao, Haohan Chi, Hanlin Wu, Zhilong Zhang, Zheng Jiang, Bingxiang He, Wei-Ying Ma, Ya-Qin Zhang, Hao Zhou
• Published: 2026-07-06
• arXiv: 2607.05394

Abstract

Reinforcement learning with verifiable rewards (RLVR) is a powerful method for improving the reasoning capabilities of language models, but it requires repeatedly generating large numbers of rollouts each time a new large model needs to be trained, making it very expensive.

This paper proposes a weak-to-strong alternative: first run RL on a smaller model (where rollout costs are lower), then transfer the learned knowledge to a larger target model.

However, directly distilling the post-RL weak teacher is not sufficient, because the teacher's final policy mixes the RL gains with the small model's inherent limitations. To address this, the authors propose Direct On-Policy Distillation (Direct-OPD), which transfers the RL-induced *policy shift* of the teacher rather than its final policy. Direct-OPD compares the post-RL teacher with its pre-RL reference version and treats the log-ratio between them as a dense implicit reward for the student.

Results

Experiments show that Direct-OPD consistently enables weak-to-strong distillation:

• Qwen3-1.7B's score on AIME 2024 improves from 48.3% to 62.4%
• Training takes only 4 hours on 8 A100 GPUs

Original Chinese Summary

可验证奖励强化学习（RLVR）是提升语言模型推理能力的强大方法，但每当需要训练新的大模型时，都要重复生成大量rollout，成本高昂。本文提出一种弱到强的替代方案：先在较小的模型上运行RL（rollout成本更低），然后将学到的知识迁移到更大的目标模型。直接蒸馏后RL的弱教师模型并不足够，因为教师最终策略混合了RL增益与小模型的局限性。为此提出Direct On-Policy Distillation（Direct-OPD），它迁移的是教师模型由RL诱导的策略偏移而非最终策略。Direct-OPD比较后RL教师与其前RL参考版本，将log-ratio视为学生的密集隐式奖励。实验表明，Direct-OPD能持续提升弱教师到强目标模型的效果：在8张A100上仅用4小时就将Qwen3-1.7B在AIME 2024上的得分从48.3%提升到62.4%。

---

*Auto-collected on 2026-07-06*