Paper: Dense Supervision, Sparse Updates: On the Sparsity and Geometry of On-Policy Distillation Authors: Guo Yu, Hao-Xuan Ma, Jun-Peng Jiang, Han-Jia Ye (School of AI, Nanjing University / State Key Laboratory for Novel Software Technology), Wenlin Liu, Yulan Hu (Alibaba AMap) Link: https://arxiv.org/abs/2606.13657
TL;DR
OPD uses dense token-level teacher supervision, yet its parameter updates are as sparse as RLVR's. What determines sparsity is not how dense the supervision is, but where the data comes from.
Background: What is OPD and why is it hot?
LLM post-training has historically had two main routes:
| Paradigm | Pros | Cons | |---|---|---| | SFT / offline distillation | Dense token-level supervision, rich signal | Fixed demonstration data; distribution shift causes error accumulation at test time | | RLVR (e.g., GRPO) | On-policy sampling avoids distribution shift | Sparse rewards; long-sequence credit assignment is hard |
On-Policy Distillation (OPD) tries to get the best of both:
- The student samples its own trajectories (on-policy, avoiding distribution shift)
- The teacher provides dense token-level supervision at every step (reverse KL divergence)
- OPD's relative update norm is only ~1% of offline distillation, slightly smaller than RLVR's.
- 66%–90% of coordinates show no visible update.
- Updates spread across layers but concentrate heavily in FFN layers; inter-layer patterns are non-uniform—some layers barely move, others update intensively.
- RLVR (sparse rewards) → sparse updates (known)
- OPD (dense supervision) → also sparse updates (new finding)
- Common factor: both sample on-policy data; only supervision density differs, yet sparsity is similar.
- SGD clearly underperforms AdamW for OPD
- Dense teacher supervision preserves heterogeneous coordinate-level gradient scales, so AdamW's adaptive scaling still helps
- Guo Yu, et al. "Dense Supervision, Sparse Updates: On the Sparsity and Geometry of On-Policy Distillation." arXiv:2606.13657, 2026.
- Mukherjee et al. (2026a). "RLVR fine-tunes small subnetworks."
- Zhu et al. (2025). "RLVR learns away from principal directions."
- Shen et al. (2026). "On the Geometry of On-Policy Distillation." arXiv:2606.07082.
This recipe has been adopted by flagship models such as Qwen3, GLM-5, MiMo, and DeepSeek-V4, becoming a third pillar of post-training pipelines. But one question has been overlooked: what does OPD actually do in parameter space?
Finding 1: Surprisingly sparse updates
Intuitively, with the teacher feeding dense signals at every step, OPD's updates should be dense like SFT. The data says otherwise:
OPD doesn't "rewrite" the model; it precisely fine-tunes a tiny subnetwork.
Practical implication: Keeping only the coordinates with significant OPD updates and freezing everything else, then retraining the subnetwork, recovers nearly identical performance to full OPD. The sparse structure is not noise—it carries the core learning task.
Finding 2: Geometry—updates live in low-dimensional corners
1. Numerically full-rank, but energy concentrated: update matrices are full-rank, yet spectral analysis shows energy concentrated in a low-dimensional subspace—updates project onto a few dominant directions. 2. Away from the source model's principal directions: using the pretrained source weight's principal singular subspace as reference, OPD updates deliberately avoid these directions, landing on coordinates where source weights are near zero—consistent with Zhu et al. (2025)'s findings on RLVR. 3. High overlap with RLVR's updated subnet: OPD's active coordinates overlap RLVR's far above random baselines—OPD is parametrically closer to RLVR than to offline distillation.
Finding 3: What determines sparsity?
The paper's most important conclusion:
> The on-policy data distribution is the core cause of update sparsity in post-training, not reward sparsity.
Dense teacher signals guide updates more precisely on sparsely activated coordinates but do not recruit more coordinates to update.
Finding 4: AdamW or SGD?
Mukherjee et al. (2026b) found RLVR works with SGD, since sparse updates need no adaptive learning rates. OPD differs:
Lesson: OPD's updates are sparse, but deciding which coordinates to update and by how much still requires adaptivity. Sparsity ≠ simplicity.
A three-dimension analysis framework
| Dimension | Question | Finding | |---|---|---| | Coordinate-level sparsity | How many coordinates update? | 66%–90% static; FFN-dominated | | Spectral concentration | Which directions hold update energy? | Full-rank but low-dimensionally dominated | | Principal-space alignment | Relation to source weight principal directions? | Deliberately avoid principal directions; favor low-magnitude coordinates |
This toolbox generalizes to analyzing any post-training method.
Why it matters
1. Fills a theory gap: OPD is widely deployed industrially, but its parameter dynamics were essentially unexplored—this is the first systematic parameter-space analysis. 2. Counterintuitive: "dense supervision = dense updates" fails. Post-training is governed by how the data distribution interacts with the model's current policy, not supervision density. 3. Efficient post-training: with 66%–90% of parameters static, memory savings and parameter-efficient OPD variants become viable—a sparse-coordinate complement to low-rank LoRA. 4. Echoes related work: Mukherjee et al. (2026a) (RLVR tunes small subnetworks), Zhu et al. (2025) (RLVR moves away from principal directions), Shen et al. (2026) "On the Geometry of On-Policy Distillation" (arXiv:2606.07082), which independently places OPD in a "relaxed off-principal regime" between SFT and RLVR.
Key numbers at a glance
| Number | Meaning | |---|---| | 66%–90% | Parameter coordinates with no visible update in OPD | | ~1% | OPD relative update norm vs. offline distillation | | ~100% | Performance recovered by training only the sparse subnet | | 3 | Analysis dimensions (coordinate sparsity, spectral concentration, principal alignment) | | 2 | Divergent optimizer conclusions: SGD suits RLVR, AdamW suits OPD |
Open questions
1. Why FFN? Do attention/FFN update differences suggest FFN stores knowledge while attention controls format? 2. Are sparse coordinates fixed? Do active subnetworks for math vs. code vs. multimodal tasks overlap—implying a universal post-training subnet? 3. Relation to LoRA? LoRA is low-rank; OPD is sparse-coordinate. Can they be combined into sparse-low-rank efficient methods? 4. Scale? Experiments cover 7B–70B; do sparsity patterns hold at frontier scale?
Conclusion
OPD is becoming a standard post-training component, but this paper reminds us: engineering intuition ≠ parameter reality. Dense supervision does not produce dense updates—the on-policy data distribution is the dominant force behind parameter sparsity. For LLM researchers, the methodological takeaway: don't only watch loss curves and benchmarks—occasionally open the parameter black box and see which coordinates move, where, and with whom.
References