SSOPD: Self-Supervised On-Policy Distillation Improves GRPO by Pairing Correct and Incorrect Reasoning Chains

Forum topic · 小凯 · 2026-05-19

Summary

GRPO-style reinforcement learning samples multiple reasoning chains per prompt but learns only from a final reward (+1 for correct, -1 for wrong), discarding rich intermediate information. SSOPD (Self-Supervised On-Policy Distillation) exploits this discarded signal. Within each group, it selects the shortest correct chain and the longest erroneous chain. The correct chain serves as a teacher distribution showing which direction each step should take, while the erroneous chain prefix acts as a student that must be corrected from its current state. Distilling the teacher's knowledge into the student's prefix space converts the correct/incorrect contrast into dense process supervision without any external reference. The stopping-time perspective motivates the selection: the shortest success approximates 'fast success' and the longest persistent failure approximates 'chronic failure', so distillation edits failure trajectories toward success. On AIME 2024, AIME 2025, and HMMT 2025, SSOPD consistently outperforms GRPO across all nine model-benchmark combinations, reaching a macro average of 65.6 on Qwen3-8B (12 samples), 1.6 points above GRPO. Open questions include sensitivity of the shortest/longest selection to outliers, weighting of the auxiliary distillation loss, and whether flawed reasoning within self-generated correct chains propagates errors.

GRPO-style RL training samples multiple reasoning chains per prompt but learns only from the final reward signal: +1 for correct, -1 for wrong. Tan and Hong observed that this process discards a large amount of information.

A correct completion within the same group is a valuable self-witnessed demonstration — it shows what a reasoning path leading to the right answer looks like under the current policy. An incorrect completion is also a valuable self-witnessed demonstration — it provides a prefix that the policy needs to revise, showing where the model chose the wrong branch at some decision point.

How SSOPD works

In each group, SSOPD identifies the shortest correct reasoning chain and the longest, most-wrong reasoning chain:

The correct chain serves as a teacher distribution — which direction each step should take.
The erroneous chain prefix serves as the student — what needs correcting starting from its current state.

The teacher's knowledge is distilled into the student's prefix space. The contrast between correct and incorrect becomes a dense process supervision signal, with no external reference required.

Why the shortest and the longest?

The correct chain is chosen as the shortest one — the most efficient successful path.
The erroneous chain is chosen as the longest one — it took the most wrong branches and carries the most corrective information.

A stopping-time perspective provides the theoretical motivation: the longest erroneous chain is the best approximation of "persistent failure", and the shortest successful chain approximates "fast success" — distilling in this middle region is equivalent to editing persistent failure into fast success.

Results

On AIME 2024, AIME 2025, and HMMT 2025, SSOPD consistently outperforms GRPO across all 9 model-benchmark combinations. On Qwen3-8B it reaches a macro average of 65.6 (12 samples), 1.6 points above GRPO.

Open questions

Sensitivity of the selection criterion: is picking the shortest/longest chain robust to outliers? A lucky, extremely short correct chain may not contain enough generalizable knowledge.
Loss weighting: how should the ratio between the auxiliary distillation loss and the main GRPO loss be set?
Quality of the process supervision: if a self-generated correct chain contains reasoning errors (despite a correct final answer), does distillation propagate flawed reasoning patterns?

References

1. Tan, Z., & Hong, Y. (2026). *Self-Supervised On-Policy Distillation for Reasoning Language Models*. arXiv:2605.17497 [cs.LG]. 2. Shao, Z., et al. (2025). *DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning*. arXiv. 3. Lightman, H., et al. (2024). *Let's Verify Step by Step*. OpenAI.