Rubric-Conditioned Self-Distillation: Fine-Grained Feedback for Reasoning LLM Post-Training

Forum topic · 小凯 · 2026-06-19

Summary

A new paper (arXiv:2506.14973) proposes Rubric-Conditioned Self-Distillation, a framework for post-training reasoning language models that replaces noisy chain-of-thought annotations and scalar reward signals with rubrics as structured, fine-grained feedback. The method conditions a teacher model on criterion-level rubrics and applies token-level guidance on the student's own sampled trajectories, enabling finer-grained credit assignment than scalar reward optimization. The framework is instantiated as a two-stage pipeline: first learning to generate task-specific rubrics, then training a rubric-guided reasoner. On diverse science reasoning benchmarks, the approach surpasses GRPO by 1.0 points and on-policy self-distillation (OPSD) by 0.9 points on average, demonstrating that rubric-level criteria can be effectively converted into token-level guidance over the reasoning process.

Paper Overview

Field: NLP Authors: Siyi Gu, Jialin Chen, Sophia Zhou Published: 2026-06-19 arXiv: 2506.14973

Background

Post-training of reasoning language models is commonly driven by two approaches, each with known weaknesses:

Supervised distillation relies on chain-of-thought annotations that are expensive to obtain and may themselves be noisy, incomplete, or partially incorrect. Even when the final solution is correct, an imperfect rationale can interfere with learning.
Reinforcement learning with verifiable rewards (RLVR) typically compresses evaluative feedback into a scalar signal, obscuring which aspects of a response should be improved.

Method

The paper proposes Rubric-Conditioned Self-Distillation, a framework that incorporates rubrics as structured, fine-grained feedback for on-policy self-distillation. The method:

1. Conditions the teacher model on criterion-level rubrics. 2. Uses the teacher to provide token-level guidance on the student's own sampled trajectories.

This design avoids treating a single reference rationale as the sole supervision target. Instead, rubrics specify what a strong response should satisfy, enabling more fine-grained credit assignment over the reasoning process than scalar reward optimization.

Pipeline

The framework is instantiated with a two-stage pipeline:

1. Learn to generate task-specific rubrics. 2. Train a rubric-guided reasoner with rubric-conditioned self-distillation.

Results

Evaluated on a diverse suite of science reasoning benchmarks, rubric-conditioned self-distillation effectively converts rubric-level criteria into token-level guidance over the reasoning process:

Surpasses GRPO by 1.0 points on average.
Surpasses OPSD by 0.9 points on average.

Original Abstract

> Post-training of reasoning language models is commonly driven by supervised distillation and reinforcement learning with verifiable rewards. Distillation often relies on chain-of-thought annotations that are expensive to obtain and may themselves be noisy, incomplete, or partially incorrect; even when the final solution is correct, an imperfect rationale can interfere with learning. Reinforcement learning with verified rewards, on the other hand, typically compresses evaluative feedback into a scalar signal, obscuring which aspects of a response should be improved. We propose Rubric-Conditioned Self-Distillation, a framework that incorporates rubrics as structured, fine-grained feedback for on-policy self-distillation. Our method conditions the teacher model on criterion-level rubrics and uses it to provide token-level guidance on the student's own sampled trajectories. This design avoids treating a single reference rationale as the sole supervision target. Instead, rubrics specify what a strong response should satisfy, enabling more fine-grained credit assignment over the reasoning process than scalar reward optimization. We instantiate this framework with a two-stage pipeline that first learns to generate task-specific rubrics and then trains a rubric-guided reasoner. We evaluate on a diverse suite of science reasoning benchmarks and results show that rubric-conditioned self-distillation effectively converts rubric-level criteria into token-level guidance over the reasoning process, surpassing GRPO by 1.0 points and OPSD by 0.9 points on average.