English static mirror for SEO/GEO · AI-assisted translation · Read Chinese original
智柴网 中文原文

Weak-to-Strong Generalization via Direct On-Policy Distillation

Forum topic · 小凯 · 2026-07-08

Summary

This paper proposes Direct On-Policy Distillation (Direct-OPD), a weak-to-strong alternative to costly Reinforcement Learning with Verifiable Rewards (RLVR) for improving LLM reasoning. Instead of running RL directly on a large model, RL is first performed on a smaller, cheaper model, then the learned gains are transferred to a larger target model. Because a weak teacher's post-RL final policy mixes RL improvements with the small model's limitations, Direct-OPD transfers the teacher's RL-induced policy shift rather than its final policy: it compares the post-RL teacher with its pre-RL reference version and treats the log-ratio as dense implicit rewards for the student. Experiments show consistent weak-to-strong gains, boosting Qwen3-1.7B's AIME 2024 score from 48.3% to 62.4% with only 4 hours of training on 8 A100 GPUs. Paper: arXiv:2607.05394.

论文概要 (Paper Overview)

Field: Machine Learning Authors: Shiyuan Feng, Huan-ang Gao, Haohan Chi, Hanlin Wu, Zhilong Zhang, Zheng Jiang, Bingxiang He, Wei-Ying Ma, Ya-Qin Zhang, Hao Zhou Published: 2026-07-06 arXiv: 2607.05394

Summary

Reinforcement Learning with Verifiable Rewards (RLVR) is a powerful method for improving the reasoning ability of language models, but it requires regenerating massive numbers of rollouts whenever a new large model needs to be trained, making it very expensive.

This paper proposes a weak-to-strong alternative: run RL on a smaller model first (where rollouts are cheaper), then transfer the learned knowledge to the larger target model.

Directly distilling the weak teacher's post-RL final policy is not sufficient, because the teacher's final policy mixes the RL gains with the small model's inherent limitations. The authors therefore propose Direct On-Policy Distillation (Direct-OPD), which transfers the teacher's RL-induced policy shift rather than its final policy. Direct-OPD compares the post-RL teacher with its pre-RL reference version and treats the log-ratio as a dense, implicit reward signal for the student.

Results

Experiments show that Direct-OPD consistently achieves weak-to-strong improvements: with only 4 hours of training on 8 A100 GPUs, it raises Qwen3-1.7B's score on AIME 2024 from 48.3% to 62.4%.

--- *Auto-collected on 2026-07-06*

Tags

#machine-learning#reinforcement-learning#distillation#llm-reasoning#weak-to-strong#rlvr#on-policy-distillation#qwen3

This page is an English static mirror generated for search and AI citation. It may be a full translation or structured summary of the Chinese original. Canonical interactive discussion lives on the Chinese page: https://zhichai.net/topic/178346214