On-Policy Self-Distillation with Sampled Demonstrations Reduces Output Diversity

Forum topic · 小凯 · 2026-06-26

Summary

This post summarizes an arXiv paper (2606.19228) by Andrei Liviu Nicolicioiu, Mohammad Pezeshki, and Aaron Courville on on-policy self-distillation in machine learning. The method uses a single model as both teacher and student, conditioning the teacher on a correct demonstration to provide dense token-level feedback, achieving strong pass@1 accuracy. The authors identify a hidden cost: rollout diversity decreases and pass@k curves flatten, so generating more rollouts no longer improves accuracy. They trace this to compounding biases in self-distillation with sampled demonstrations—the teacher scores each student rollout while conditioned on a sampled correct rollout, channeling feedback through the model's own biases. Theoretical analysis shows the optimal self-distillation policy tilts the base distribution via a pointwise conditional mutual information score, and unlike ideal on-policy RL, it can amplify existing probability gaps, concentrating mass on already dominant modes. Experiments on a controlled graph pathfinding task and science QA benchmarks show self-distillation matches or exceeds RL on average performance, but exhibits significantly lower functional and semantic diversity and fails in out-of-distribution settings requiring diverse strategies.

Paper Overview

Field: Machine Learning Authors: Andrei Liviu Nicolicioiu, Mohammad Pezeshki, Aaron Courville arXiv: 2606.19228

Abstract

On-policy self-distillation achieves strong pass@1 accuracy by using a single model as both teacher and student, with the teacher conditioned on a correct demonstration to provide dense token-level feedback. We show that this could come at a hidden cost: rollout diversity decreases and pass@k curves flatten (i.e., generating more rollouts fails to improve accuracy). We trace this to compounding biases in the design of self-distillation with sampled demonstrations. The teacher scores each student rollout while conditioned on a sampled correct rollout, channeling its feedback through the model's own biases.

We theoretically analyze the optimal self-distillation policy and show that it tilts the base distribution by a pointwise conditional mutual information score between the student's rollout and the correct demonstration used as context. Unlike an ideal on-policy reinforcement learning (RL) setup—which preserves probability ratios among equally correct rollouts—self-distillation can amplify existing probability gaps, concentrating mass on modes that already dominate.

Key Findings

On-policy self-distillation improves pass@1 accuracy but silently reduces rollout diversity.
Pass@k curves flatten: sampling more rollouts no longer improves accuracy.
The problem stems from compounded bias when the teacher is conditioned on sampled correct demonstrations.
The optimal self-distillation policy tilts the base distribution via a pointwise conditional mutual information score, amplifying existing probability gaps.
Experiments on a controlled graph pathfinding task and science QA benchmarks show self-distillation matches or exceeds RL on average performance.
However, distilled models show significantly lower functional and semantic diversity and fail in out-of-distribution settings that require diverse strategies.

--- *Auto-collected on 2026-06-26*