Self-Policy Distillation: AI Teaching Itself Only the Right Capabilities, Up to 16% Gain Without External Signals

Forum topic · 小凯 · 2026-05-22

Summary

Researchers at the University of Cambridge propose Self-Policy Distillation (SPD), a self-distillation method for large language models that requires no external signals such as verifiers, reward models, or execution feedback. Conventional self-distillation suffers from confirmation bias: models reinforce their own errors, style preferences, and formatting habits when trained on their own outputs. SPD addresses this in two steps. First, it extracts a low-rank capability subspace via SVD over gradients computed on correctness-defining tokens (e.g., final answers or key code outputs) using a small calibration set. Second, during self-generation, the model's Key-Value activations are projected onto this subspace, filtering out errors and stylistic noise while preserving task competence. The method is self-policy: the generating and trained models are the same. Across code generation, math reasoning, and multiple-choice QA, using five LLM backbones (1.5B–14B) and six datasets, SPD improves up to 13% over the strongest external-signal-free self-distillation baseline, up to 16% over pretrained baselines, and 15% on cross-domain generalization. Notably, projected generation shows a self-correction effect, producing more accurate answers even without verification. Limitations include reliance on identifiable correctness tokens and minor projection overhead, but SPD suggests internal self-filtering as a path for frontier model self-improvement.

Self-Policy Distillation: Letting AI Teach Itself — But Only the "Right" Capabilities

When a student self-studies, the biggest danger is not failing to learn, but memorizing their own mistakes as correct answers — drifting further off track the more they study.

Self-distillation for large language models faces exactly the same problem. Having a model generate answers and then train on them sounds great, but the model absorbs its own errors, style preferences, and formatting habits — like a student treating their wrong answers as the standard.

A team from the University of Cambridge proposes an elegant solution: Self-Policy Distillation (SPD) — with no external signals whatsoever, the model itself can distinguish "which capabilities to learn, and which to discard."

The Triple Dilemma of Self-Distillation

First, understand why existing self-distillation methods fall short. Imagine you are a model that must generate its own training data. You have three options:

Option 1: Filter with external signals. Use verifiers to check code correctness, reward models for scoring, or execution feedback. Problem: these signals are expensive, and for the strongest frontier models, you may not find any better "judge."

Option 2: No filtering — accept everything. Train directly on all self-generated outputs. Simple and crude, but the model reinforces its own errors — "confirmation bias" — becoming more stubborn with training.

Option 3: SPD — internal filtering, no external judge. This paper's approach, and the only one satisfying both "no external signals" and "capability selectivity."

Core Idea: Extracting a "Capability Subspace" from Gradients

SPD's key insight: the model's internal activation patterns differ between when it gets answers right and wrong. From this difference, we can extract a low-dimensional subspace where "capability" lives.

Concretely, SPD has two steps:

Step 1: Extract the capability subspace. Using a small calibration set (e.g., a few hundred problems), compute gradients on the model's "correctness-defining tokens." What are these? For math problems, the digits of the final answer; for code, the key outputs. SVD (singular value decomposition) is then applied to these gradients to extract a low-rank projection matrix — the "capability subspace."

Step 2: Self-generate within the capability subspace. When generating training data, the model's Key-Value activations are projected onto this subspace. It's like fitting the model with a "capability filter" — only signals relevant to correct capability pass through, while style preferences, formatting habits, and common errors are filtered out. The model is then fine-tuned on these "filtered" outputs.

A key detail: SPD uses self-policy distillation, not off-policy. The data-generating model and the trained model are the same — not "an old model teaching a new model," but "a model teaching itself, learning only the right parts."

Experimental Results: Leading Across the Board

SPD was tested in three domains — code generation, math reasoning, and multiple-choice QA — across five LLM backbones (1.5B to 14B parameters) and six datasets.

vs. the strongest external-signal-free self-distillation method: up to 13% improvement
vs. pretrained baselines: up to 16% improvement
Cross-domain generalization: 15% better than baselines — the most striking number

Why is generalization so strong? Because SPD's extracted "capability subspace" captures general reasoning ability, not task-specific surface patterns. When the model generates data within this subspace, it naturally produces more generalizable reasoning paths.

An Interesting Finding: "Self-Correction" After Projection

The researchers observed an unexpected phenomenon: when the model's KV activations are projected onto the capability subspace, generation quality improves noticeably — even without external verification, the model tends to produce more correct answers.

It's like someone thinking in a noisy room where every idea pops up; but in a quiet space that keeps only core thinking, ideas become clearer. The projection essentially creates a "quiet thinking space" for the model.

Limitations and Outlook

SPD currently relies on annotating "correctness-defining tokens" — easy for math and code (answers and outputs), but less direct for open-ended tasks like creative writing. Projection also introduces minor computational overhead — small, but worth considering at very large deployment scale.

Still, SPD's philosophical significance may outweigh its technical details: it demonstrates that the model's internals already contain the information to distinguish "right" from "wrong" — no external judge needed, just the right "perspective" to extract it. This has profound implications for frontier model self-evolution: as models grow stronger and external signals become harder to obtain, internal self-filtering may be the only way forward.

---

Paper link: Self-Policy Distillation via Capability-Selective Subspace Projection