Apple's Simple Self-Distillation (SSD): Zero Labels, Zero Verification, Zero Teacher — Yet Code Performance Jumps 30%

Forum topic · 小凯 · 2026-05-06

Summary

Apple researchers propose SSD (Simple Self-Distillation), an embarrassingly simple post-training method for code generation. A frozen model samples one solution per problem at a chosen temperature, with no answer verification, no reference labels, no teacher model, and no reinforcement learning. Standard supervised fine-tuning on these self-generated outputs lifts Qwen3-30B-Instruct on LiveCodeBench v6 from 42.4% to 55.3% pass@1 (+30.4% relative), with the largest gains on hard problems. Pass@5 improves even more, showing SSD preserves rather than compresses exploration diversity. The authors explain the mechanism via a Precision-Exploration Conflict: some token positions (locks) need sharp precision while others (forks) need diverse exploration, and no single decoding temperature can serve both. SSD reshapes the distribution context-dependently. A key finding is an effective temperature law: T_eff = T_train × T_eval, with an optimum near 1.2. Caveats raised in the analysis include heavy infrastructure needs (8x B200 GPUs), unverified long-run stability, and cross-domain stability only tested on 30B models. Paper: arXiv 2604.01193; code at github.com/apple/ml-ssd.

Simple Self-Distillation (SSD): A Model Can Teach Itself With Its Own Wrong Answers — and Get Better at Code

> Paper: Embarrassingly Simple Self-Distillation Improves Code Generation > Authors: Ruixiang Zhang*, Richard He Bai*, Huangjie Zheng*, Navdeep Jaitly, Ronan Collobert, Yizhe Zhang* > Institution: Apple > arXiv: 2604.01193v1 > GitHub: https://github.com/apple/ml-ssd

1. A Counterintuitive Starting Point

A model trains on its own unverified answers — and becomes smarter. Apple's SSD method is almost suspiciously simple:

1. Give the model a coding problem 2. Sample one solution at a specific temperature — no correctness verification 3. Standard supervised fine-tuning (SFT) on that raw output 4. Test

No human reference answers, no stronger teacher, no reward model, no code execution, no RL. Yet Qwen3-30B-Instruct jumps from 42.4% to 55.3% pass@1 on LiveCodeBench v6 — a 30.4% relative gain, largest on the hardest problems. All five tested models (Llama, Qwen; 4B/8B/30B; instruct/thinking) improve.

2. How Simple Is "Simple"?

Data synthesis: for a frozen model p_θ, sample y ~ Decode_{T_train, ρ_train}(p_θ(·|x)). Only N=1 sample per problem. No verification — just minimal syntax filtering (drop empty replies and one-line stubs).

Training: standard cross-entropy SFT. No KL constraint, no contrastive loss, no RL gradients.

Inference: the fine-tuned model decodes with a *different* configuration (T_eval, ρ_eval). Crucially, train and inference temperatures compose via a discoverable mathematical relation.

3. The Numbers

LiveCodeBench v6 (pass@1):

| Model | Base | +SSD | Gain | |---|---|---|---| | Qwen3-30B-Instruct | 42.4% | 55.3% | +12.9pp (+30.4% rel.) | | Llama-3.1-8B-Instruct | baseline | — | +3.5pp | | Qwen3-4B-Instruct | baseline | — | +7.5pp | | Qwen3-4B-Thinking | baseline | — | +3.3pp | | Qwen3-30B-Thinking | baseline | — | +2.1pp |

By difficulty (Qwen3-30B-Instruct): Easy +6.5pp, Medium +14.2pp, Hard +15.3pp. The harder the problem, the bigger the gain.

Pass@5 (exploration diversity): Overall +18.1pp; Hard +23.0pp. SSD does not compress diversity — it widens the exploration space while sharpening precision.

4. The Core Mechanism: The Precision-Exploration Conflict

Code has two kinds of token positions:

Lock positions: syntax/semantics force a specific value, but a long tail of plausible-yet-wrong distractors still competes for probability. These need precision.
Fork positions: multiple genuinely valid paths (loop vs. recursion vs. data-structure init) lead to different algorithms. These need exploration.

A single global decoding temperature must compromise: low T sharpens locks but flattens forks; high T restores forks but revives distractor tails. The paper names this the Precision-Exploration Conflict.

How SSD resolves it

Support compression: high-temperature sampling exposes edge tokens; fine-tuning sharpens lock positions and suppresses distractor tails.
Within-support reshaping: at forks, diverse sampled paths teach the model to keep useful options balanced.

SSD reshapes the distribution context-dependently — more precise where precision is needed, more open where exploration is needed.

5. The Algebra of Temperature

The paper finds a composable rule: effective temperature T_eff = T_train × T_eval.

Without truncation, T_eff almost fully governs performance (R² = 0.75), peaking near T_eff ≈ 1.2
T_train and T_eval trade off: high-train + low-eval ≈ low-train + high-eval
Adding truncation (top-k=10) raises the ceiling further by cutting low-probability tails during synthesis

Why isn't just tuning the temperature enough?

Sweeping T_eval on the base model moves pass@1 only from 41.3% to 43.5% (a 2.2pp range) vs. SSD's 55.3% — an 11.8pp gap, even larger on hard problems (+13.3pp pass@1, +19.4pp pass@5). SSD permanently changes the model's internal distribution shape; it cannot be replicated by inference-time tuning.

6. A Critical Look: Five Hard Questions

1. "Embarrassingly Simple" algorithm, "Embarrassingly Expensive" infrastructure. Experiments used 8× B200 GPUs, Megatron-LM, and 65,536 sequence length. Algorithmic simplicity ≠ experimental accessibility. 2. The cost of N=1. If the single sampled solution is wrong, the model actively learns its own mistake. The paper doesn't quantify the error rate in training data or its impact. 3. Why doesn't SSD collapse? Unsupervised methods like majority voting and entropy minimization suffer reward hacking; SSD doesn't — but only within 2500 iterations (instruct) / 300 (thinking). Long-run behavior is untested. 4. Cross-domain stability is only verified on 30B models. The paper's claim of stable math/general performance tested only 30B; smaller models may lack the capacity to absorb domain shift. 5. "Zero teacher" is really an internalized teacher. The base model's own distribution serves as its teacher — not external, but not conjured from nothing.

7. The Bigger Picture

SSD complements — not replaces — human labeling, teacher distillation, and RL as a post-training path. It's a free baseline when you lack verification environments, stronger teachers, or RL infrastructure.

The key insight: existing code models contain unrealized capability locked away by fixed decoding configurations. SSD unlocks built-in knowledge rather than injecting new knowledge.

The lock/fork framework may generalize: math proofs (computation = lock, strategy = fork), creative writing (plot turns = forks), dialogue systems (fact retrieval = locks, style = forks).

8. Bottom Line

A model can train on its own wrong answers and get smarter — by learning *where* to be precise and where to explore, not what is correct.
T_eff = T_train × T_eval, optimal ≈ 1.2; truncation raises the ceiling further.
"Simple" applies to the algorithm, not the infrastructure.
Cross-domain stability is only validated at 30B scale.
SSD is a complementary, zero-cost post-training baseline alongside RL.

References

Zhang, R., Bai, R. H., Zheng, H., Jaitly, N., Collobert, R., & Zhang, Y. (2026). Embarrassingly Simple Self-Distillation Improves Code Generation. arXiv:2604.01193v1.
GitHub: https://github.com/apple/ml-ssd
HuggingFace models: apple/SimpleSD-4B-instruct, apple/SimpleSD-4B-thinking, apple/SimpleSD-30b-a3b-instruct