Trust Region Policy Distillation: When a Teacher's Unbounded Logit Punishments Drive the Student Model Crazy

Forum topic · 小凯 · 2026-10-19

Summary

On-policy distillation lets a 30B teacher model score a 8B student's tokens during reinforcement learning, but it breaks down: the log-ratio reward becomes unboundedly negative when the student emits tokens the teacher considers near-impossible, exploding gradients and destabilizing training. A paper testing this pipeline found a Qwen3-8B student distilled from a Qwen3-30B teacher scored 24.58 on AIME24, worse than a student trained with direct RLVR (30.10). The proposed fix, Trust Region Policy Distillation, mixes the teacher with the student itself to form a 'proximal teacher': alpha times the teacher probability plus (1-alpha) times the student probability. This caps punishment at log(alpha), roughly -1.2, and mathematically reduces to a one-line change in the log-ratio reward with no extra compute. The authors prove bounded gradient variance, global convergence, and monotonic improvement guarantees. Results: AIME24 jumps from 24.58 to 50.42, beating the direct RL baseline by 20 points, with gains across six math benchmarks. The full pipeline is reproducible on a single 8-GPU node; validation is limited to an 8B student.

Trust Region Policy Distillation: When a Teacher's Scoring Drives the Student Crazy

Having a 30B model teach an 8B model sounds great in theory: the teacher can solve more problems, the student learns by imitation, no human labels or reward model needed — just hand a math problem to both models. This route has a formal name: on-policy distillation. A paper tested it last month, and the results were awkward: after learning from the 30B teacher, the 8B student scored 24.58 on the AIME math competition. A student that skipped the teacher and did direct reinforcement learning scored 30.10.

Learning from the teacher was five points worse than not learning at all. Where does it break?

The Crack in the Scoring

Here's how on-policy distillation works: the student solves problems on its own, and the teacher scores every token. Each time the student writes a token, the teacher outputs "my probability for this token," which is compared against the student's own probability; the log of the ratio serves as reward or penalty — if the teacher thinks the token makes sense, bonus; if the teacher finds it absurd, penalty.

The crack is in the denominator. If the teacher's probability is 0.0001 and the student writes that token anyway, the penalty is the log of the ratio of the two probabilities — a near-zero number divided by a normal number, then logged — and the penalty races toward negative infinity. The gradient explodes and training collapses.

This isn't a rare scenario; it's the everyday scenario. The student can't solve the problems — that's why it's learning. It frequently produces tokens the teacher considers "nearly impossible." The stricter the teacher, the more easily the student gets knocked out by a single negative infinity. The whole training pipeline is mathematically unstable. The paper's own words: "this unbounded penalty makes the policy gradient extremely fragile."

A One-Line Formula

The fix is comically simple: dilute the teacher.

Instead of using the pure teacher probability as the reference, use 70% teacher plus 30% student — the paper calls this mixture the "proximal teacher." The formula is one line: teacher probability times alpha, plus student probability times (1 minus alpha).

The effect is immediate. When the student writes an absurd token, the proximal teacher's penalty now has a floor: the reference contains 30% of the student itself, so the student's token has at least 30% probability under the mixed reference, capping the punishment at log(0.3), roughly -1.2. Negative infinity becomes -1.2.

There's more elegance downstream. You don't actually need to construct this mixed teacher — no need to compute and average both models' probabilities before every scoring step. After algebraic simplification of the objective, it reduces to rewriting the original reward in one line: log of (alpha times the probability ratio plus (1 minus alpha)). The paper's phrasing: "thanks to the math." Conceptually you've added a teacher; computationally, not a single extra token of work. Setting alpha to 1 degenerates back to vanilla distillation — the new method strictly contains the old one, which is just an extreme-parameter special case.

The Numbers

After this one-line change, with the same student-teacher pair, AIME24 goes from 24.58 to 50.42 — more than doubled. AIME26 goes from 25.42 to 44.06. All six math benchmarks improve; the smallest gain is 3.25 on MATH-500. Against direct RL, it's 50.42 vs 30.10 — a 20-point lead: a good teacher really helps, as long as the teacher doesn't beat the student into madness.

The theory is also a highlight: the authors prove the mixed teacher bounds gradient variance by construction, with a global convergence proof and a monotonic improvement bound — every training step is guaranteed non-regressive. This machinery is classic RL orthodoxy: "trust region" comes from TRPO (2015), whose message is "don't take big update steps." That paper, by OpenAI's Schulman, limits the magnitude of policy updates; this one shifts the same idea from "how far each update goes" to "how far the teacher itself sits from the student." A teacher too far away is out of the student's reach; mix in 30% of the student, and the teacher is within arm's length.

One practical detail: the entire training pipeline is fully reproducible on a single 8-GPU node, per the paper. The student is the Qwen3-8B base model, the teacher is a Qwen3-30B instruct model, same family — big teaching small, strong teaching weak, no cross-family confounders. Limitations stated too: validation only goes up to an 8B student; larger students are untested.

Looking back at that awkward opening comparison: distillation losing to direct RL wasn't because the teacher was useless, but because a good teacher's scoring method drives the student insane. Mix 30% of the student into the teacher, same model pair, same compute, and scores double. Sometimes what's stuck in a field isn't resources — it's a symbol: that log racing toward negative infinity, missing a (1 - alpha) underneath.

---

*Source: arXiv 2607.04751v1 (2026-07-06), "Trust Region Policy Distillation"; authors Zhengpeng Xie (HKUST Guangzhou), Li Lyna Zhang (Microsoft Research Asia), Zeke Xie (corresponding, HKUST Guangzhou), Mao Yang (Microsoft); student Qwen3-8B-Base, teacher Qwen3-30B-A3B-Instruct-2507; AIME24/25/26, AMC23, MATH-500, OlympiadBench figures all from Table 2 of the paper; RLVR baselines are GRPO/DAPO.*