Self-Policy Distillation: Letting AI Teach Itself—But Only the "Right Capabilities"—with 16% Gains and No External Signals
What does a self-studying student fear most? Not failing to learn, but memorizing their own mistakes as if they were correct answers—drifting further off course with each study session.
Large language model self-distillation faces exactly the same problem. Having a model generate answers and then train on them sounds great, but the model learns its own errors, style preferences, and formatting habits along the way—like a student treating wrong answers as standard solutions.
A team at the University of Cambridge has proposed an elegant solution: Self-Policy Distillation (SPD)—with no external signals at all, the model itself can distinguish "which capabilities to keep and which to discard."
The Three-Way Dilemma of Self-Distillation
First, understand why existing self-distillation methods fall short. Imagine you are a model that must generate its own training data. You have three options:
Option 1: Filter with external signals. For example, use verifiers to check code correctness, reward models to score outputs, or execution feedback to judge results. The problem: these signals are expensive, and for the strongest frontier models, you may not find any "judge" better than the model itself.
Option 2: Accept everything, no filtering. Train directly on all self-generated outputs. Simple and blunt, but the model reinforces its own errors—this is called "confirmation bias," and training only makes it more stubborn.
Option 3: SPD—internal filtering, no outsiders needed. This is the paper's approach, and the only one that satisfies both "no external signals required" and "capability selectivity."
Core Idea: Extracting a "Capability Subspace" from Gradients
SPD's key insight: the model's internal activation patterns differ when it gets things right versus wrong. From this difference, we can extract a low-dimensional subspace where "capability" lives.
Concretely, SPD works in two steps:
Step 1: Extract the capability subspace. Using a small calibration set (e.g., a few hundred problems), compute the model's gradients on "correctness-defining tokens." What are those? For math problems, the digits of the final answer; for code problems, the key outputs. Then apply SVD (singular value decomposition) to these gradients to extract a low-rank projection matrix—the "capability subspace."
Step 2: Self-generate within the capability subspace. When generating training data, project the model's Key-Value activations onto this subspace. It's like putting a "capability filter" on the model—only signals related to correct capability pass through, while style preferences, formatting habits, and common errors are filtered out. The model is then fine-tuned on these filtered outputs.
One crucial detail: SPD uses self-policy distillation, not off-policy distillation. The model generating the data and the model being trained are the same—not an "old model teaching a new model," but a model teaching itself while learning only the correct parts.
Experimental Results: Leading Across the Board
SPD was tested in three domains—code generation, math reasoning, and multiple-choice QA—across five LLM backbones (from 1.5B to 14B parameters) and six datasets.
- Versus the strongest external-signal-free self-distillation method: up to 13% improvement
- Versus pretraining baselines: up to 16% improvement
- Cross-domain generalization: 15% better than baselines—the most striking number
An Interesting Finding: Projected "Self-Correction"
The researchers observed an unexpected phenomenon: when the model's KV activations are projected onto the capability subspace, the quality of its generated outputs improves markedly—even without external verification, the model tends to produce more correct answers.
It's like someone thinking in a noisy room where every kind of thought pops up; give them a quiet space that retains only core thinking, and their ideas become clearer. The projection essentially creates a "quiet thinking space" for the model.
Limitations and Outlook
SPD currently relies on labeling "correctness-defining tokens"—easy for math and code (answers and outputs), but less straightforward for open-ended tasks like creative writing. The projection also introduces a small computational overhead that matters at very large scale.
Still, SPD's philosophical significance may outweigh its technical details: it shows that the model's internals already contain information distinguishing "right" from "wrong." We don't need external judges—we just need the right "perspective" to extract that information. This has deep implications for the self-evolution of frontier models: when models grow strong enough and external signals become harder to obtain, internal self-filtering may be the only path forward.
---
Paper link: Self-Policy Distillation via Capability-Selective Subspace Projection