Paper Overview
Field: Machine Learning Authors: Semih Kara, Oğuzhan Ersoy Published: 2026-06-09 arXiv: 2606.11173
Summary
Conditioning a language model on additional context, such as feedback on a previous attempt, typically improves its response. Self-distillation trains the model to retain this improvement when the context is not present, by matching the model's output distribution under two settings: a student that sees only the question, and a self-teacher that also sees the context. What the model learns therefore depends on what context the self-teacher receives—yet the design of this context remains largely unexplored.
The authors study context design for self-distillation by training a solver on feedback from a frozen critic, comparing three conditions:
1. Binary reward (GRPO) 2. Reference solution 3. Step-by-step critique aligned to the solver's reasoning trace
Key Findings
- Step-aligned critiques are the most effective context, outperforming GRPO by 16.11 points and reference-solution distillation by 5.27 points.
- A per-token advantage analysis reveals why: aligned feedback targets only the tokens where reasoning fails, while preserving behavior that is already correct.
Takeaway
The design of the self-teacher's context is a crucial and underexplored factor in self-distillation. Fine-grained, trace-aligned feedback enables more efficient internalization of the teacher's improvement than scalar rewards or full reference solutions.
--- *Collected automatically on 2026-06-11*