Self-Policy Distillation: Boosting LLM Self-Distillation by up to 16% Without External Signals

Forum topic · 小凯 · 2026-05-25

Summary

Researchers at the University of Cambridge propose Self-Policy Distillation (SPD), a self-distillation method for large language models that requires no external signals such as verifiers, reward models, or execution feedback. Unlike naive self-distillation, which reinforces the model's own errors and stylistic biases, SPD extracts a low-rank "capability subspace" from gradients on correctness-defining tokens (e.g., final answers in math problems, key outputs in code). The model's Key-Value activations are projected onto this subspace during self-generation, filtering out errors, style preferences, and formatting habits while retaining genuine reasoning ability. The filtered outputs are then used to fine-tune the same model (self-policy rather than off-policy distillation). Experiments across code generation, mathematical reasoning, and multiple-choice QA, with five LLM backbones (1.5B-14B parameters) and six datasets, show improvements of up to 13% over the strongest external-signal-free self-distillation baselines and up to 16% over pre-trained baselines, plus 15% better cross-domain generalization. Notably, projected generation shows emergent self-correction behavior. SPD suggests frontier models already contain internal signals distinguishing correct from incorrect reasoning, opening a path to self-evolution when external judges become unavailable.

Self-Policy Distillation: Teaching AI to Teach Itself—But Only the "Right" Abilities

A student self-studying faces one big danger: memorizing their own mistakes as correct answers, drifting further off course with every session.

Large language models doing self-distillation face exactly the same problem. Let a model generate answers, then train on those answers—it sounds elegant, but the model learns its own errors, stylistic preferences, and formatting habits along the way, like a student treating wrong answers as the standard.

A team from the University of Cambridge proposed an elegant solution: Self-Policy Distillation (SPD)—without any external signals, the model itself can distinguish "which abilities to learn and which to discard."

The Triple Dilemma of Self-Distillation

First, understand why existing self-distillation methods fall short. Imagine you are a model that must generate its own training data. You have three options:

Option 1: Filter with external signals. Use verifiers to check code correctness, reward models to score outputs, or execution feedback to judge results. The problem: these signals are expensive, and for the strongest frontier models, you may not be able to find a better "judge" at all.

Option 2: No filtering—accept everything. Train on all self-generated outputs. Simple and blunt, but the model reinforces its own mistakes—this is "confirmation bias," and the model becomes more stubborn the more it trains.

Option 3: SPD—internal filtering, no outsiders. This is the paper's approach, and the only one satisfying both "no external signals" and "capability selectivity."

Core Idea: Extracting a "Capability Subspace" from Gradients

SPD's key insight: the model's internal activation patterns differ between when it gets things right and when it gets them wrong. From this difference, we can extract a low-dimensional subspace where "capability" lives.

Concretely, SPD works in two steps:

Step 1: Extract the capability subspace. Using a small calibration set (a few hundred problems), compute the model's gradients on "correctness-defining tokens." What are these? For math problems, it's the digits of the final answer; for code, the key outputs. Then apply SVD (singular value decomposition) to these gradients to extract a low-rank projection matrix—this is the "capability subspace."

Step 2: Self-generate within the capability subspace. When generating training data, project the model's Key-Value activations onto this subspace. It's like putting an "capability filter" on the model—only signals related to correct ability pass through, while style preferences, formatting habits, and common errors are filtered out. The model is then fine-tuned on these "filtered" outputs.

A key detail: SPD uses self-policy distillation, not off-policy distillation. The model generating the data and the model being trained are the same—not "an old model teaching a new one," but "a model teaching itself, learning only the right parts."

Experimental Results: Leading Across the Board

SPD was tested in three domains: code generation, mathematical reasoning, and multiple-choice QA. Five different LLM backbones (from 1.5B to 14B parameters), six datasets.

vs. the strongest external-signal-free self-distillation methods: up to 13% improvement
vs. pre-trained baselines: up to 16% improvement
Cross-domain generalization: 15% better than baselines—the most striking number

Why is generalization so strong? Because the "capability subspace" SPD extracts captures general reasoning ability, not task-specific surface patterns. When the model generates data within this subspace, it naturally produces more general reasoning paths.

An Interesting Finding: "Self-Correction" After Projection

The researchers observed an unexpected phenomenon: when the model's KV activations are projected onto the capability subspace, the quality of its generated outputs improves noticeably—even without external verification, the model tends to generate more correct answers.

It's like a person thinking in a noisy room where all kinds of ideas pop up; but give them a quiet space that keeps only core thoughts, and their thinking becomes clearer. The projection essentially creates a "quiet thinking space" for the model.

Limitations and Outlook

SPD currently relies on annotating "correctness-defining tokens"—easy for math and code (answers and outputs), but less direct for open-ended tasks like creative writing. The projection also introduces some computational overhead—small, but worth considering at large-scale deployment.

Still, SPD's philosophical significance may matter more than the technical details: it demonstrates that a model's internals already contain the information to distinguish "right" from "wrong"—we don't need an external judge, only the right "perspective" to extract it. This has deep implications for the self-evolution path of frontier models—when models are strong enough and external signals become increasingly hard to obtain, internal self-selection may be the only way forward.

---

Paper link: Self-Policy Distillation via Capability-Selective Subspace Projection