Self-Distilled RLVR: Using Self-Distillation as GRPO's Credit Assignment Officer
> Paper: Self-Distilled RLVR (arXiv:2604.03128) > Authors: Chenxu Yang, Chuanyu Qin, Qingyi Si, et al. > Institutions: Institute of Information Engineering, CAS; University of Chinese Academy of Sciences; Microsoft Research Asia; JD Group > Link: https://arxiv.org/abs/2604.03128
1. The Problem: GRPO's "Uniform Payroll" Reward
GRPO suffers from unclear credit assignment. For a reasoning chain of hundreds of tokens, the verifier only returns a 0/1 sequence-level signal, meaning:
- The key reasoning step leading to a correct answer gets the same reward weight as the trailing "Therefore, the answer is..." boilerplate
- In a wrong answer, the actual erroneous step is penalized equally with correct earlier steps
- Frequency of referencing "reference solution" monotonically increases during training (leakage)
- Validation accuracy peaks at steps 10–20, then continuously declines
- Teacher–student KL divergence stagnates after initial decline
L*: ideal marginal-matching objective (optimizable)I(Y_t; R | X, Y_{<t}): a conditional mutual information term — an irreducible gap independent of parameters θ- A > 0 (correct): teacher-supported tokens get larger weights
- A < 0 (wrong): teacher-opposed tokens get larger penalties
- RLSD achieves the best average, +4.69 over base and +2.32 over GRPO; math gains are largest (MathVision +3.91)
- OPSD and SDPO both underperform GRPO, validating the leakage theory; naive GRPO+OPSD combination also fails
- Training dynamics: RLSD converges faster and higher; GRPO shows rapid entropy collapse while RLSD retains higher entropy; clip ratio stays at 3%–6%
- Token-level visualizations confirm sensible credit: key steps (identifying critical objects, final subtraction) get high credit; generic narration gets little; in wrong trajectories, the actual misreading gets the largest penalty
- Validated only on Qwen3-VL-8B; generalization to other architectures (e.g., text-only LLMs) unverified
- Privileged information limited to ground-truth answers; richer forms (intermediate reasoning steps) unexplored
- Absolute overhead at very large scale still to be evaluated
- Follow-up arXiv:2605.10781 (Rebellious Student) inverts RLSD's weighting for exploration; the framework may extend to multi-turn dialogue and tool-use chains
This sequence-level sparse signal undermines convergence for long chain-of-thought reasoning.
2. The OPSD Trap: Why On-Policy Self-Distillation Collapses Late in Training
OPSD design: the same model acts as both teacher and student; the teacher receives privileged information (e.g., the reference answer) and provides dense per-token logits supervision.
Observed pathologies (Figure 3):
Example leaked outputs: *"Given that the reference solution uses 9 values, I will assume..."*, *"But wait, the reference solution says 'No', which contradicts my calculation..."*
Theoretical diagnosis (Theorem 1):
OPSD forces a conditionally-independent student to match a conditionally-dependent teacher — an ill-posed problem. Gradient analysis (Proposition 1): g(θ; r) = g*(θ) + δ(θ; r), where δ is an r-specific bias whose variance grows with the teacher's reliance on privileged information.
Two-phase dynamics: 1. Early: ‖g*‖ ≫ ‖δ‖ — fast learning of general reasoning, accuracy rises 2. Late: as P_S → P̄_T, g* → 0 and δ dominates — the model encodes spurious x → r correlations and leakage erupts
3. RLSD: Decoupling Direction and Magnitude
> "Update direction and update magnitude have asymmetric requirements."
| Signal | Requirement | Source | |---|---|---| | Direction | Sparse but reliable | Environment reward (verifier) | | Magnitude | Denser is better | Self-distillation evidence ratio |
Three-step algorithm
Step 1 — Privileged information gain (stop-gradient, weights only):
Step 2 — Sign-aware evidence reweighting:
Bayesian interpretation: the ratio is a Bayesian evidence ratio measuring how much generating y_t makes r consistent with the trajectory.
Since exp(·) > 0, w_t never flips the sign of the advantage — the environment reward retains exclusive control of update direction; the teacher only modulates magnitude.
Step 3 — Clipped credit assignment:
with ε_w = 0.2, plus a λ schedule decaying from 0.5 to 0 over the first 50 steps.
Overhead: only one extra forward pass per response — negligible compared to rollout generation.
4. Experiments: Multimodal Reasoning on Qwen3-VL-8B
Setup: Qwen3-VL-8B-Instruct; MMFineReason-123K difficulty-filtered subset (samples where Qwen3-VL-4B-Thinking failed all 4 rollouts); 4 nodes × 8 × NVIDIA H200 140GB.
| Method | MMMU | MathVista | MathVision | ZeroBench | WeMath | Avg | |---|---:|---:|---:|---:|---:|---:| | Base LLM | 62.44 | 73.80 | 47.37 | 19.76 | 54.10 | 51.49 | | GRPO | 65.11 | 76.20 | 48.82 | 22.60 | 56.57 | 53.86 | | OPSD | 63.82 | 75.10 | 47.53 | 21.06 | 54.95 | 52.49 | | SDPO | 65.11 | 74.00 | 47.27 | 25.15 | 52.19 | 52.74 | | GRPO+OPSD | 63.22 | 75.90 | 48.52 | 22.16 | 54.76 | 52.91 | | RLSD (Ours) | 67.22 | 78.10 | 52.73 | 24.85 | 58.00 | 56.18 |
Key findings:
5. Why RLSD Does Not Leak
| Method | Token-level advantage | Direction | Magnitude | |---|---|---|---| | GRPO | A (uniform) | Environment reward | None (uniform) | | OPSD | Δ_t | Teacher preference | Teacher preference | | RLSD | A · clip(w_t, ...) | Environment reward | Teacher evidence ratio |
RLSD satisfies all three conditions of the impossibility trilemma (Appendix A.6): objective stability, sustained improvement, and zero-leakage training — by moving P^T/P^S from gradient direction to gradient magnitude.
6. Assessment: Elegant Recombination, Not a New Paradigm
RLSD invents no new components: GRPO's sequence-level advantage, OPSD's self-distillation signal, and PPO-style clipping are all retained. It merely reassigns their roles — the environment reward remains the judge of correctness, while self-distillation becomes the recorder that determines how much reward or punishment each token receives. Distillation acts as RL's co-pilot: direction stays with the verifier, magnitude gets finer-grained.