On-Policy Self-Distillation with Sampled Demonstrations Reduces Output Diversity

Forum topic · 小凯 · 2026-06-26

Summary

Researchers Andrei Liviu Nicolicioiu, Mohammad Pezeshki, and Aaron Courville show that on-policy self-distillation, where a single model acts as both teacher and student and the teacher is conditioned on a sampled correct demonstration to give dense token-level feedback, achieves strong pass@1 accuracy at a hidden cost: rollout diversity drops and pass@k curves flatten, so sampling more rollouts no longer improves accuracy. The authors trace this to compounding biases in self-distillation with sampled demonstrations: the teacher scores each student rollout while conditioned on a sampled correct rollout, channeling feedback through the model's own biases. They theoretically analyze the optimal self-distillation policy, showing it tilts the base distribution by a pointwise conditional mutual information score between the student's rollout and the demonstration used as context. Unlike ideal optimal on-policy RL, which preserves probability ratios among equally correct rollouts, self-distillation can amplify existing probability gaps and concentrate mass on already dominant modes. Experiments on a controlled graph pathfinding task and scientific QA benchmarks show self-distilled models match or exceed RL in average performance but exhibit significantly lower functional and semantic diversity and fail in out-of-distribution settings that require diverse strategies. Paper: arXiv 2606.19228.

Paper Overview

Field: Machine Learning
Authors: Andrei Liviu Nicolicioiu, Mohammad Pezeshki, Aaron Courville
arXiv: 2606.19228

Abstract (translated)

On-policy self-distillation achieves strong pass@1 accuracy by using a single model as both teacher and student, with the teacher conditioned on a correct demonstration to provide dense token-level feedback. We show that this could come at a hidden cost: rollout diversity decreases and pass@k curves flatten (i.e., generating more rollouts fails to improve accuracy).

We trace this to compounding biases in the design of self-distillation with sampled demonstrations. The teacher scores each student rollout while conditioned on a sampled correct rollout, channeling its feedback through the model's own biases.

We theoretically analyze the optimal self-distillation policy and show that it tilts the base distribution by a pointwise conditional mutual information score between the student's rollout and the correct rollout used as context. Unlike ideal optimal on-policy reinforcement learning (RL), which preserves probability ratios among equally correct rollouts, self-distillation can amplify existing probability gaps, concentrating mass on already dominant modes.

On a controlled graph pathfinding task and scientific QA benchmarks, self-distilled models match or exceed RL in average performance, but exhibit significantly lower functional and semantic diversity, failing in out-of-distribution settings that require diverse strategies.

Key Takeaways

Self-distillation with sampled demonstrations boosts pass@1 but flattens pass@k curves.
The root cause is compounding bias: the teacher conditions on a sampled correct rollout and routes feedback through the model's own biases.
Theoretically, the optimal self-distillation policy reweights by a pointwise conditional mutual information score, favoring dominant modes.
Empirically, self-distillation matches RL on average accuracy but sacrifices functional/semantic diversity and OOD robustness.

--- *Auto-collected on 2026-06-26*