Self-Distilled RLVR: Letting Self-Distillation Serve as GRPO's "Credit Assigner"

> Paper: Self-Distilled RLVR > Authors: Chenxu Yang, Chuanyu Qin, Qingyi Si, et al. > Affiliations: Institute of Information Engineering, CAS; University of Chinese Academy of Sciences; Microsoft Research Asia; JD.com > Link: https://arxiv.org/abs/2604.03128

1. The Problem: GRPO's "Equal Pay" Reward

GRPO has a chronic flaw: unclear credit assignment. A rollout of hundreds of tokens receives a single 0/1 verifier reward, meaning:

• The key reasoning step that produced the correct answer gets the same reward weight as the trailing "Therefore, the answer is..." filler.
• In a wrong answer, the actually erroneous step is penalized equally with the correct setup before it.

This sequence-level sparse signal severely limits convergence efficiency for long chain-of-thought reasoning.

2. The OPSD Trap: Why an Elegant Fix Fails Late in Training

2.1 OPSD Design

• The same model acts as both teacher and student.
• The teacher receives privileged information (e.g., the ground-truth answer or reference solution).
• The teacher evaluates each token of the student's output, providing dense logits-level supervision.

2.2 Odd Training Dynamics (Figure 3)

| Symptom | Description | |:---|:---| | Leak frequency | Quoting the "reference solution" rises monotonically during training | | Validation accuracy | Peaks at steps 10–20, then declines continuously | | Teacher–student KL | Briefly drops, then plateaus at a fixed level |

Typical leaked outputs (Figure 2): > "Given that the reference solution uses 9 values, I will assume..." > "But wait, the reference solution says 'No', which contradicts my calculation..."

2.3 Theoretical Diagnosis

The paper proves (Theorem 1) that the OPSD objective decomposes as:

\[L_{OPSD} = L_* + I(Y_t; R \mid X, Y_{<t})\]
• L_*: the ideal marginal-matching objective (optimizable)
• I(Y_t; R | X, Y_{<t}): a conditional mutual information — an irreducible gap independent of parameters θ

Core contradiction: OPSD forces the student (conditionally independent parameterization) to match the teacher (conditionally dependent distribution) — an ill-posed problem.

Gradient-level diagnosis (Proposition 1):

\[g(\theta; r) = g_*(\theta) + \delta(\theta; r)\]
• g_*: benign marginal-matching gradient
• δ: r-specific bias whose variance scales with the teacher's reliance on privileged information

Two-phase dynamics: 1. Early: ‖g_*‖ ≫ ‖δ‖ — the model quickly learns general reasoning; accuracy rises steeply. 2. Late: as \(P_S \to \bar{P}_T\), \(g_* \to 0\); the bias δ dominates — the model starts encoding "x → r" correlations and leakage explodes.

3. RLSD: Decoupling Direction from Magnitude

3.1 Core Insight

> "Update direction and update magnitude have asymmetric requirements."

| Signal | Requirement | Source | |:---|:---|:---| | Direction | Sparse but reliable | Environment reward (verifier) | | Magnitude | As dense as possible | Self-distillation evidence ratio |

3.2 Three-Step Algorithm

Step 1: Privileged information gain

\[\Delta_t = \mathrm{sg}(\log P^T(y_t) - \log P^S(y_t))\]
• Same model, two input conditions; sg = stop-gradient, so Δ_t is purely a weighting signal.

Step 2: Direction-aware evidence reweighting

\[w_t = \exp(\mathrm{sign}(A) \cdot \Delta_t) = (P^T(y_t) / P^S(y_t))^{\mathrm{sign}(A)}\]

Bayesian interpretation: P^S is the prior belief (given only x), P^T the posterior (after observing privileged information r); the ratio measures how much y_t makes r consistent with the trajectory.

• A > 0 (correct): w_t = P^T/P^S → teacher-supported tokens get larger weight
• A < 0 (wrong): w_t = P^S/P^T → teacher-opposed tokens get larger penalty

> Key guarantee: exp(·) > 0 ensures w_t > 0, so sign(Â_t) = sign(A). > The environment reward retains exclusive control of update direction; the teacher only modulates magnitude.

Step 3: Clipped credit assignment

\[\hat{A}_t = A \cdot \mathrm{clip}(w_t, 1-\epsilon_w, 1+\epsilon_w)\]
• ε_w = 0.2 caps each token's maximum influence
• Interpolation λ ∈ [0,1] decays from 0.5 to 0 over the first 50 steps

3.3 Full Objective

\[L_{RLSD}(\theta) = E\left[\frac{1}{G}\sum_i \frac{1}{|y^{(i)}|}\sum_t \min(w_t A^{(i)}, \mathrm{clip}(w_t,\dots) A^{(i)})\right]\]

Compute overhead: only 1 extra forward pass per answer — negligible versus rollout generation.

4. Experiments: Multimodal Reasoning on Qwen3-VL-8B

4.1 Setup

• Base model: Qwen3-VL-8B-Instruct
• Data: MMFineReason-123K (difficulty-filtered subset where Qwen3-VL-4B-Thinking failed all 4 rollouts)
• Hardware: 4 nodes × 8 × NVIDIA H200 140GB

4.2 Results Across Five Benchmarks (Table 2)

| Method | MMMU | MathVista | MathVision | ZeroBench | WeMath | Avg | |:---|---:|---:|---:|---:|---:|---:| | Base LLM | 62.44 | 73.80 | 47.37 | 19.76 | 54.10 | 51.49 | | GRPO | 65.11 | 76.20 | 48.82 | 22.60 | 56.57 | 53.86 | | OPSD | 63.82 | 75.10 | 47.53 | 21.06 | 54.95 | 52.49 | | SDPO | 65.11 | 74.00 | 47.27 | 25.15 | 52.19 | 52.74 | | GRPO+OPSD | 63.22 | 75.90 | 48.52 | 22.16 | 54.76 | 52.91 | | RLSD (Ours) | 67.22 | 78.10 | 52.73 | 24.85 | 58.00 | 56.18 |

Key findings:

• RLSD achieves the best average accuracy: +4.69 over the base LLM and +2.32 over GRPO.
• Math reasoning improves most: MathVista +1.9, MathVision +3.91.
• Both OPSD and SDPO underperform GRPO, confirming the leakage analysis.
• The naive GRPO+OPSD combination fails (+3.27 gap vs RLSD), showing simple addition doesn't work.

4.3 Training Dynamics (Figure 5)

| Metric | Observation | Implication | |:---|:---|:---| | Training reward | RLSD rises faster and converges higher; OPSD collapses late | Faster convergence + stability | | Training entropy | GRPO suffers rapid entropy collapse; RLSD maintains higher entropy | Selective reinforcement without suppressing exploration | | Clip ratio | Stable at 3%–6% | Trust region works as intended |

4.4 Token-Level Credit Visualization (Figure 6)

• Correct trajectory (cube counting): high credit (green) on identifying the key yellow cubes and the final subtraction; low credit (red) on generic narration like "Looking at the image, I see..."
• Wrong trajectory (bar chart): high penalty (deep red) on the misread relation "3x=28.5" and wrong answer "x=9.5"; low penalty on neutral setup tokens.

5. Why RLSD Doesn't Leak

| Method | Token-level advantage | Direction source | Magnitude source | |:---|:---|:---|:---| | GRPO | A (uniform) | Environment reward | None (uniform) | | OPSD | Δ_t = log P^T − log P^S | Teacher preference | Teacher preference | | RLSD | A · clip(w_t, ...) | Environment reward | Teacher evidence ratio |

RLSD satisfies all three conditions of the impossibility triangle (Appendix A.6):

• ✅ Objective stability
• ✅ Sustained improvement
• ✅ Zero-leakage training

The core mechanism: moving P^T/P^S from gradient direction to gradient magnitude. Since exp(·) > 0, w_t never flips the sign of the update direction.

6. Verdict: Not a New Paradigm, but an Elegant Stitching

RLSD invents no new components:

• GRPO's sequence-level advantage? Kept.
• OPSD's self-distillation signal? Kept.
• PPO's clipping? Kept.

It merely redistributes their roles: the environment reward stays the judge deciding right vs. wrong; self-distillation stops being a judge and becomes the clerk recording how much reward or penalty each token deserves. Distillation becomes RL's co-pilot — the driver (direction) is still the environment reward, while the co-pilot (magnitude) helps see more finely.

7. Limitations and Extensions

Limitations: 1. Validated only on Qwen3-VL-8B; generalization to other architectures (e.g., text-only LLMs) remains unverified. 2. Privileged information is limited to the ground-truth answer; richer forms (e.g., intermediate reasoning steps) unexplored. 3. The extra forward pass is negligible at this scale but should be re-evaluated for very large models.

Follow-up work:

• arXiv:2605.10781 (Rebellious Student) inverts RLSD's weighting mechanism for exploration.
• The framework extends to other domains needing fine-grained credit assignment (multi-turn dialogue, tool-use chains).