Paper Overview
- Field: NLP
- Authors: Siyi Gu, Jialin Chen, Sophia Zhou
- Published: 2026-06-19
- arXiv: 2506.14973
- Surpasses GRPO by 1.0 points on average
- Surpasses OPSD by 0.9 points on average
Abstract
Post-training of reasoning language models is commonly driven by supervised distillation and reinforcement learning with verifiable rewards. Distillation often relies on chain-of-thought annotations that are expensive to obtain and may themselves be noisy, incomplete, or partially incorrect; even when the final solution is correct, an imperfect rationale can interfere with learning. Reinforcement learning with verified rewards, on the other hand, typically compresses evaluative feedback into a scalar signal, obscuring which aspects of a response should be improved.
We propose Rubric-Conditioned Self-Distillation, a framework that incorporates rubrics as structured, fine-grained feedback for on-policy self-distillation. Our method conditions the teacher model on criterion-level rubrics and uses it to provide token-level guidance on the student's own sampled trajectories. This design avoids treating a single reference rationale as the sole supervision target. Instead, rubrics specify what a strong response should satisfy, enabling more fine-grained credit assignment over the reasoning process than scalar reward optimization.
Method
The framework is instantiated with a two-stage pipeline:
1. Rubric generation: first learn to generate task-specific rubrics. 2. Rubric-guided reasoning: then train a rubric-guided reasoner, where the teacher provides token-level guidance on the student's own sampled trajectories conditioned on criterion-level rubrics.
Results
Evaluated on a diverse suite of science reasoning benchmarks, rubric-conditioned self-distillation effectively converts rubric-level criteria into token-level guidance over the reasoning process: