Self-Policy Distillation: Boosting LLM Self-Distillation by 16% Without External Signals

Forum topic · 小凯 · 2026-05-22

Summary

Researchers from the University of Cambridge propose Self-Policy Distillation (SPD), a self-distillation method that improves large language models without any external reward signals, verifiers, or execution feedback. Conventional self-distillation risks confirmation bias, as models reinforce their own errors, stylistic preferences, and formatting habits. SPD solves this by extracting a low-rank 'capability subspace' from gradients on correctness-decisive tokens (e.g., final answers in math, key outputs in code) via SVD, using a small calibration set. During training-data generation, the model's KV activations are projected onto this subspace, filtering out errors and style while preserving genuine reasoning ability. Experiments across code generation, mathematical reasoning, and multiple-choice QA on five LLM backbones (1.5B–14B parameters) and six datasets show gains of up to 13% over the strongest external-signal-free self-distillation baseline, up to 16% over pretrained baselines, and 15% better cross-domain generalization. A notable finding: projecting activations produces 'self-correction,' yielding higher-quality outputs even without external validation. Limitations include reliance on identifiable correctness tokens and minor projection overhead.

Self-Policy Distillation: Teaching AI Only the 'Right' Capabilities — 16% Gain Without External Signals

What does a self-taught student fear most? Not failing to learn, but memorizing their own mistakes as correct answers — drifting further off course the more they study.

Self-distillation of large language models (LLMs) faces exactly the same problem. Have a model generate answers, then train on those answers — it sounds great, but the model absorbs its own errors, stylistic preferences, and formatting habits along the way, like a student treating wrong answers as the standard.

A team from the University of Cambridge proposes an elegant solution: Self-Policy Distillation (SPD) — requiring no external signals at all, the model itself can distinguish which capabilities to learn and which to discard.

The Three-Way Dilemma of Self-Distillation

First, understand why existing self-distillation methods fall short. Imagine you are a model generating training data to improve yourself. You have three options:

Option 1: Filter with external signals. Use verifiers to check code correctness, reward models to score outputs, or execution feedback to judge results. The problem: these external signals are expensive, and for the strongest frontier models, you may not be able to find a better 'judge' at all.

Option 2: No filtering — accept everything. Train directly on all self-generated outputs. Simple and blunt, but the model reinforces its own errors — this is 'confirmation bias,' and the model becomes more entrenched the longer it trains.

Option 3: SPD — internal filtering, no outside help. This is the paper's approach, and the only one that satisfies both 'no external signals' and 'capability selectivity.'

Core Idea: Extracting the 'Capability Subspace' from Gradients

SPD's key insight: a model's internal activation patterns differ when it gets things right versus wrong. From this difference, we can extract a low-dimensional subspace where 'capability' lives.

Concretely, SPD works in two steps:

Step 1: Extract the capability subspace. Using a small calibration set (e.g., a few hundred problems), compute the model's gradients on 'correctness-decisive tokens.' What are those? For math problems, the digits of the final answer; for code problems, the key outputs. Then apply SVD (singular value decomposition) to these gradients to extract a low-rank projection matrix — the 'capability subspace.'

Step 2: Self-generate within the capability subspace. When generating training data, project the model's Key-Value activations onto this subspace. It's like putting a 'capability filter' on the model — only signals related to correct capability pass through; stylistic preferences, formatting habits, and common errors are filtered out. The model is then fine-tuned on these 'filtered' outputs.

A crucial detail: SPD uses self-policy distillation, not off-policy distillation. That means the model generating the data and the model being trained are the same one — not 'an old model teaching a new model,' but 'a model teaching itself, learning only the right parts.'

Experimental Results: Comprehensively Ahead

SPD was tested across three domains — code generation, mathematical reasoning, and multiple-choice QA — with five different LLM backbones (from 1.5B to 14B parameters) on six datasets.

vs. the strongest external-signal-free self-distillation method: up to 13% improvement
vs. pretrained baselines: up to 16% improvement
Cross-domain generalization: 15% better than baselines — the most striking number

Why is generalization so strong? Because the 'capability subspace' SPD extracts captures general reasoning ability, not task-specific surface patterns. When the model generates data within this subspace, it naturally produces more generalizable reasoning paths.

An Interesting Discovery: 'Self-Correction' After Projection

The researchers found something unexpected: when the model's KV activations are projected onto the capability subspace, the quality of its generated outputs improves markedly — even without external validation, the model tends to produce more correct answers.

It's like a person thinking in a noisy room where all kinds of ideas pop up; give them a quiet space that retains only core thinking, and their ideas become clearer. Projection essentially creates a 'quiet thinking space' for the model.

Limitations and Outlook

SPD currently relies on annotating 'correctness-defining tokens' — easy for math and code (answers and outputs), but less straightforward for open-ended tasks like creative writing. The projection also introduces a small computational overhead — not much, but worth considering for very large-scale deployments.

Still, SPD's philosophical significance may outweigh its technical details: it demonstrates that a model's internals already contain the information needed to distinguish 'right' from 'wrong.' We don't need an external judge — only the right 'perspective' to extract that information. This has profound implications for the self-evolution path of frontier models: when models are strong enough and external signals become increasingly hard to obtain, internal self-selection may be the only way forward.

---

Paper link: Self-Policy Distillation via Capability-Selective Subspace Projection