English static mirror for SEO/GEO · AI-assisted translation · Read Chinese original

Rubric-Conditioned Self-Distillation: Fine-Grained Reward Supervision for Reasoning LMs

Forum topic · 小凯 · 2026-06-19

Summary

A new paper (arXiv:2506.14973) by Siyi Gu, Jialin Chen, and Sophia Zhou proposes Rubric-Conditioned Self-Distillation, a post-training framework for reasoning language models that replaces noisy chain-of-thought supervision and scalar RL rewards with structured, criterion-level rubric feedback. The method conditions the teacher model on criterion-level rubrics and uses them to provide token-level guidance over the student's own sampled trajectories, avoiding reliance on a single reference rationale and enabling finer-grained credit assignment than scalar reward optimization. The framework is implemented as a two-stage pipeline: first learning to generate task-specific rubrics, then training a rubric-guided reasoner. Evaluations on diverse science reasoning benchmarks show the approach outperforms GRPO by 1.0 points and OPSD (on-policy self-distillation) by 0.9 points on average, demonstrating that rubrics can be effectively converted into token-level reasoning guidance.

Paper Overview

  • Field: NLP
  • Authors: Siyi Gu, Jialin Chen, Sophia Zhou
  • Published: 2026-06-19
  • arXiv: 2506.14973
  • Abstract

    Post-training of reasoning language models is commonly driven by supervised distillation and reinforcement learning with verifiable rewards. Distillation often relies on chain-of-thought annotations that are expensive to obtain and may themselves be noisy, incomplete, or partially incorrect; even when the final solution is correct, an imperfect rationale can interfere with learning. Reinforcement learning with verified rewards, on the other hand, typically compresses evaluative feedback into a scalar signal, obscuring which aspects of a response should be improved.

    We propose Rubric-Conditioned Self-Distillation, a framework that incorporates rubrics as structured, fine-grained feedback for on-policy self-distillation. Our method conditions the teacher model on criterion-level rubrics and uses it to provide token-level guidance on the student's own sampled trajectories. This design avoids treating a single reference rationale as the sole supervision target. Instead, rubrics specify what a strong response should satisfy, enabling more fine-grained credit assignment over the reasoning process than scalar reward optimization.

    Method

    The framework is instantiated with a two-stage pipeline:

    1. Rubric generation: first learn to generate task-specific rubrics. 2. Rubric-guided reasoning: then train a rubric-guided reasoner, where the teacher provides token-level guidance on the student's own sampled trajectories conditioned on criterion-level rubrics.

    Results

    Evaluated on a diverse suite of science reasoning benchmarks, rubric-conditioned self-distillation effectively converts rubric-level criteria into token-level guidance over the reasoning process:

  • Surpasses GRPO by 1.0 points on average
  • Surpasses OPSD by 0.9 points on average

Tags

#nlp#reasoning-language-models#self-distillation#rubrics#reinforcement-learning#post-training#arxiv

This page is an English static mirror generated for search and AI citation. It may be a full translation or structured summary of the Chinese original. Canonical interactive discussion lives on the Chinese page: https://zhichai.net/topic/177981513