Anti-Self-Distillation: When AI Refuses to Copy Its Own Correct Answers, Reasoning Speed Jumps 10x

Forum topic · 小凯 · 2026-05-24

Summary

Self-distillation in reasoning RL can backfire: when a model learns from its own correct answers, it overfits to conclusive filler while discarding exploratory 'deliberation tokens' like 'wait' and 'let me think' — a failure mode called Teacher Confidence Inflation. The AntiSD (Anti-Self-Distillation) algorithm reverses the usual KL-divergence objective, deliberately pushing the student away from the answer-conditioned teacher, and uses Pointwise Mutual Information (PMI) analysis to identify genuinely decisive tokens, plus an entropy trigger that switches off the distillation signal when the teacher collapses into overconfident mode. Reported results across five models from 4B to 30B parameters show 2x–10x training speedups and up to 11.5-point gains on math benchmarks, with deeper, more deliberate reasoning that converges in hundreds of steps instead of thousands. This post explains the mechanism, the inverted optimization formula, and what it means for teacher-free self-improvement of LLMs.

Anti-Self-Distillation: Rejecting Your Own Correct Answers Makes Reasoning 10x Faster

The trap of self-taught intelligence

A common trick for teaching LLMs math reasoning is "standing on your own left foot": let the model solve problems, and if it gets them right, distill its chain-of-thought (CoT) back into itself — self-distillation. But this method has a strange pathology: sometimes the model improves, sometimes it gets *worse*, even when learning from verified correct answers.

It resembles a student who memorizes answers: present a slightly modified exam question, and the rote logic falls apart.

The root cause: spoiled by the answer

Digging deeper, researchers found the problem. When the model learns while looking at the correct answer, it develops an illusion: it eagerly learns conclusive, connective filler text, while skipping the tentative, exploratory words that actually crack problems — phrases like "wait", "let me think", "maybe".

> This is called Teacher Confidence Inflation: the correct answer acts as a spoiler, so the model only replicates the result and loses the "mulling over" quality of the derivation process.

The fix: AntiSD's contrarian approach

The AntiSD (Anti-Self-Distillation) algorithm flips the script:

1. Reverse pull: instead of forcing the student to cling to the answer-conditioned teacher, it maintains a deliberate "centrifugal force" between them. 2. Pointwise Mutual Information (PMI) analysis: precisely identifies which tokens are filler and which are decisive. 3. Entropy trigger: when the teacher starts collapsing into overconfident mode (entropy collapse), the distillation signal is switched off and a robust mode kicks in.

The core objective inverts the usual KL minimization:

\nabla_\theta \mathcal{L}_{AntiSD} = - \eta \cdot \nabla_\theta \mathbb{D}_{KL}(\pi_{teacher} || \pi_{student})

> Note the minus sign: traditional learning descends the KL divergence, while AntiSD deliberately ascends it — keeping the student at a distance so it retains independent reasoning.

Results

Researchers ran AntiSD on five models ranging from 4B to 30B parameters, and it won across the board — logic that previously took thousands of training steps now emerges in hundreds.

This means an LLM can finally "rescue itself": with just the correct answers and this contrarian algorithm, it pulls itself out of its own logical mud — no stronger teacher required.

Reference

Paper: Anti-Self-Distillation for Reasoning RL via Pointwise Mutual Information
Released: May 20, 2026
arXiv: arXiv:2605.11609
Problem addressed: loss of "Deliberation Tokens" and logical degradation caused by self-distillation in reasoning RL
Mechanism: the AntiSD strategy, which uses PMI analysis to identify and strengthen key decision points in the reasoning process