LOPD: Latent On-Policy Self-Distillation

> Paper: *Latent On-Policy Self-Distillation* — arXiv: 2608.13040v1 (submitted 2026-08-13) > Authors: Guibin Zhang (SJTU/NUS), Jiayang Lyu, Ran Sun, Xinlei Yu, Haoyu Zhao, Qibing Ren†, Shuicheng Yan† > Code: github.com/bingreeky/LOPD (Apache-2.0; inference code released, training code "coming soon") > Checkpoints: Qwen3-8B-LOPD (tool calling), OLMo-3-7B-Think-LOPD (code generation)

Key Points

• Core idea: Prior self-distillation methods require a human-designed "privileged cheat sheet" (gold answers, expert traces, extracted skills) given to the teacher. LOPD instead lets this cheat sheet emerge from experience as a set of learned continuous latent variables, trained end-to-end. After training, the student runs alone — no latents, retrieval, or teacher — for zero inference overhead.
• Four-step pipeline:
1. Retrieval: an experience bank B stores only successful rollouts (passing reward thresholds or all tests), embedded with Qwen3-Embedding-8B (4096-dim), FAISS inner-product index, top-3 retrieval (n_ret=3), stored in observation-lite format. 2. Latent composition: a QFormer-style module (encoder LoRA rank 8, 8 shared-weight cross-attention layers with learnable queries) compresses each retrieved trajectory into K=32 continuous latent tokens — 96 tokens total for 3 experiences. 3. Teacher conditioning: the student sees only the current state; a frozen backbone copy of the teacher additionally receives the latent tokens injected at a <|LATENT_PH|> placeholder position (supported by SGLang/vLLM). Teacher and student share the same prefix. 4. Distillation: reverse-KL dense distillation on the student's own trajectories, plus a privileged-margin objective. Cold start: one SFT round on successful trajectories with the backbone frozen, training only LoRA/QFormer.
• The privileged-margin objective (the key trick): a per-token privilege measure δ = log π^T − sg[log π^S], averaged with trajectory-level outcome weighting A(τ)=2r(τ)−1. The total objective is a Lagrangian min-max: distillation loss + β·(m − Δ(φ)) with margin m=0.05, forcing the teacher to stay at least 0.05 nats ahead of the student (otherwise the teacher collapses into a copy of the student). An anchoring term (λ=0.2) keeps latents near their cold-start values; dual variable updated by β ← [β + η_β(m − Δ)]_+ with η_β=0.5.

Results

Tool calling (0–100, Qwen3-4B / 8B):

| Method | EnvScaler | BFCL-v3 | ACEBench | |---|---|---|---| | Vanilla | 48.6 / 49.2 | 22.88 / 28.38 | 50.6 / 54.6 | | GRPO | 61.8 / 57.3 | 25.25 / 29.00 | 56.0 / 58.0 | | Skill-SD | 59.1 / 60.2 | 24.63 / 27.38 | 56.0 / 56.0 | | OPSD | 51.2 / 52.0 | 25.13 / 25.75 | 48.6 / 52.7 | | LOPD | 63.7 / 66.4 | 27.38 / 29.88 | 60.6 / 62.7 |

Code generation (pass@1 %): LOPD achieves 48.78 (Qwen3-4B, LiveCodeBench avg) vs 48.29 for GRPO; 50.98 (OLMo3-7B) vs 48.29 GRPO; HumanEval+ 90.24 for OLMo3-7B-LOPD.

Behavioral internalization (EnvScaler, Qwen3-4B): reward 0.486 → 0.637; tool calls per step 3.50 → 1.11; first-step length 9,937 → 6,210 tokens — evidence that expert behavior is internalized into the policy rather than looked up at inference.

Ablations: freezing the composer gives 0.573; joint training with m=0 drops to 0.551 (teacher collapse); m=0.05 peaks at 0.637 (0.10→0.626, 0.20→0.613). Latent capacity sweet spot at K=32; retrieval n_ret=3 optimal.

Critical Assessment

1. Possible test-set tuning: margin m, capacity K, and n_ret were all swept on the same test set (EnvScaler), so headline numbers may not transfer. 2. The "<30% rollout budget" claim is inflated: Figure 4 shows LOPD reaching 0.637 at 576/1600 generations — about 36%, not 30%. 3. No error bars or multi-seed runs — all tables report single-point numbers. 4. Latents are a black box: projecting the 32 latent tokens yields fragmented multilingual/code-like text; compact but uninterpretable and non-interventionable. 5. Cold start depends on the base model producing successful rollouts — it cannot bootstrap from zero. 6. Limited domains: only tool calling and code generation tested.

Positioning vs. Related Work

• OPSD → SDPO → Skill-SD represent progressively smarter human/relation-specified privileged formats (fixed answers → environment feedback → LLM-summarized text skills). LOPD eliminates the cheat sheet entirely — a paradigm shift rather than an incremental tweak.
• GRPO/RLVR: outcome-based RL; strong where correctness is a natural signal, weaker in multi-turn agent settings with sparse rewards. LOPD claims to outperform RLVR.

Practical Notes

• The repo (interaction/, envs/, backends/, memory/, inference/, utils/, configs/) is a usable engineering scaffold for adding learnable experience memory to agents (ReAct runner, SGLang/vLLM backends, QFormer + FAISS + compressor in memory/).
• Inference is genuinely zero-overhead (student-only deployment), but training code is unreleased — self-training requires waiting or re-implementing the QFormer injection.

One-Page Summary

• LOPD = the teacher's cheat sheet grows itself from experience; no hand design.
• Four steps: retrieve → compress to 96 latent tokens → condition a frozen teacher → reverse-KL distill with privileged margin.
• The margin (m=0.05) is the soul of the method: without it the teacher collapses into the student.
• Wins on both battlefields, though margins are modest; the real highlight is efficiency (~36%, not the claimed 30%).
• Zero inference overhead; production friendly. Training code pending — checkpoints only for now.