SSOPD: Self-Supervised On-Policy Distillation Turns GRPO's Failed Reasoning Chains into Dense Process Supervision

Forum topic · 小凯 · 2026-05-19

Summary

GRPO-style RL training samples multiple reasoning chains per prompt but learns only from a final binary reward (+1 correct, -1 wrong), discarding rich information contained in the sampled chains. SSOPD (Self-Supervised On-Policy Distillation), proposed by Tan and Hong, exploits this untapped signal. Within each sampling group, it selects the shortest correct chain as a teacher distribution—an example of an efficient successful reasoning path—and the longest erroneous chain as a student prefix containing the most wrong-branch decisions to correct. Distilling the teacher into the student's prefix space converts the correct-vs-wrong contrast into dense process supervision without any external reference. The stop-time perspective motivates this: the longest failing chain approximates persistent failure, the shortest success approximates fast success, and distillation between them edits failure into success. On AIME 2024, AIME 2025, and HMMT 2025, SSOPD consistently outperforms GRPO across all nine model-benchmark combinations, reaching a macro-average of 65.6 on Qwen3-8B (12 samples), 1.6 points above GRPO. Open questions include sensitivity of the shortest/longest selection to outliers, weighting of the auxiliary distillation loss versus the GRPO objective, and whether flawed reasoning in self-generated correct chains could propagate bad reasoning patterns.

GRPO-style RL training samples multiple reasoning chains per prompt but only learns from the final reward signal—+1 for a correct answer, -1 for a wrong one. Tan and Hong noticed that this process discards a large amount of information.

Correct completions within the same group are valuable self-witnesses: they show what a reasoning path leading to the right answer looks like under the current policy. Wrong completions are also valuable self-witnesses: they provide prefixes the policy needs to fix, marking decision points where the model chose the wrong branch.

How SSOPD works

In each group, SSOPD identifies the shortest correct reasoning chain and the longest most-erroneous chain:

The correct chain serves as a teacher distribution—showing which direction each step should take.
The erroneous chain prefix serves as the student—starting from the current state, showing what needs to be corrected.

The teacher's knowledge is distilled into the student's prefix space. The contrast between correct and wrong becomes a dense process supervision signal, with no external reference required.

Why shortest-correct and longest-wrong?

The correct chain is chosen as the shortest—the most efficient successful path.
The erroneous chain is chosen as the longest—it took the most wrong branches and contains the most corrective information.

The stopping-time perspective provides theoretical motivation: the longest failing chain is the best approximation of "persistent failure," while the shortest successful chain approximates "fast success." Distilling in this intermediate region is equivalent to editing persistent failure toward fast success.

Results

On AIME 2024, AIME 2025, and HMMT 2025, SSOPD consistently outperforms GRPO across all 9 model–benchmark combinations. On Qwen3-8B it reaches a macro-average of 65.6 (12 samples), 1.6 points above GRPO.

Open questions

Is the shortest/longest selection criterion sensitive to outliers—a lucky extremely short correct chain may not contain enough generalizable knowledge?
How should the distillation loss weight be set—what is the ratio between the auxiliary loss and the GRPO main loss?
What is the quality of the process supervision signal—if a self-generated correct chain contains reasoning errors (despite a correct answer), does distillation propagate flawed reasoning patterns?

References

1. Tan, Z., & Hong, Y. (2026). *Self-Supervised On-Policy Distillation for Reasoning Language Models*. arXiv:2605.17497 [cs.LG]. 2. Shao, Z., et al. (2025). *DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning*. arXiv. 3. Lightman, H., et al. (2024). *Let's Verify Step by Step*. OpenAI.