Dense Supervision, Sparse Updates: Dissecting the Parameter Dynamics of On-Policy Distillation (OPD)

> Paper: Dense Supervision, Sparse Updates: On the Sparsity and Geometry of On-Policy Distillation > Authors: Guo Yu, Hao-Xuan Ma, Jun-Peng Jiang, Han-Jia Ye (Nanjing University, School of AI / State Key Lab for Novel Software Technology), Wenlin Liu, Yulan Hu (Alibaba Amap) > Link: https://arxiv.org/abs/2606.13657

TL;DR

OPD uses dense token-level teacher supervision, yet its parameter updates are as sparse as RLVR's. What determines sparsity is not "how dense the supervision is" but "where the data comes from."

Background: What is OPD and why did it take off?

LLM post-training has largely followed two paradigms:

| Paradigm | Pros | Cons | |----------|------|------| | SFT / offline distillation | Dense token-level supervision, rich signal | Fixed demonstration data causes distribution shift and error accumulation at test time | | RLVR (e.g., GRPO) | On-policy sampling avoids distribution shift | Sparse rewards, hard credit assignment over long sequences |

OPD (On-Policy Distillation) tries to get both:

• The student samples its own trajectories (on-policy, no distribution shift)
• A teacher provides dense token-level supervision at every step (via reverse KL divergence)

This recipe has been adopted by flagship models (Qwen3, GLM-5, MiMo, DeepSeek-V4) and is becoming the third pillar of post-training pipelines. Yet one question has been largely ignored: what does OPD actually do in parameter space?

Finding 1: Surprisingly sparse updates

Intuition says OPD's dense supervision should produce dense, SFT-like weight rewrites. The data disagrees:

Extremely small update magnitude

OPD's weight update relative norm is only ~1% of offline distillation's, slightly smaller than RLVR's.

Striking coordinate-level sparsity

| Metric | Result | |--------|--------| | Fraction of coordinates with no visible update | 66% ~ 90% | | Update distribution | Spread across layers, but dominated by FFN layers | | Cross-layer pattern | Non-uniform — some layers barely move, others update intensively |

Despite dense per-token supervision, most parameter coordinates barely move. OPD doesn't "rewrite" the model — it precisely fine-tunes a tiny subnetwork.

Practical implication: a sparse subnetwork recovers full performance

In an extreme experiment, the authors kept only coordinates with significant OPD updates, froze everything else, and retrained the subnetwork: performance was nearly identical to full OPD. The sparse structure is not noise — it's an effective subnetwork carrying the core learning signal.

Finding 2: Geometry — updates hide in low-dimensional corners

1. Numerically full-rank, but energy-concentrated

• Update matrices are numerically full-rank
• Spectral analysis shows energy concentrated in a low-dimensional subspace — updates project onto a few dominant directions

2. Away from the source model's principal directions

Using the pretrained source model's principal singular subspace as reference, OPD's updates deliberately avoid these directions, landing on coordinates where source weights are near zero. This matches Zhu et al. (2025) on RLVR: post-training doesn't touch the core pretrained knowledge — it fills in "empty corners."

3. High overlap with RLVR's active subnetwork

OPD's active coordinates overlap RLVR's far above random baselines — confirming OPD's parameter dynamics are closer to RLVR than to offline distillation.

Finding 3: What determines sparsity?

The paper's most important conclusion:

> The on-policy data distribution is the core driver of update sparsity in post-training — not reward sparsity.

• RLVR (sparse rewards) → sparse updates (known)
• OPD (dense supervision) → also sparse updates (new)
• Common factor: both use on-policy samples; the difference is supervision density, yet sparsity is similar

Dense teacher supervision only guides the sparse active coordinates more precisely — it doesn't recruit more coordinates to update.

Finding 4: AdamW or SGD?

Mukherjee et al. (2026b) found RLVR works even with SGD, since sparse updates don't need adaptive learning rates. OPD is different:

• SGD clearly underperforms AdamW in OPD
• Reason: dense teacher supervision preserves heterogeneous coordinate-wise gradient scales; adaptive scaling still matters

Takeaway: sparse updates ≠ simple updates — both coordinate selection and per-coordinate scale need adaptation.

A three-axis analysis framework

| Axis | Question | Finding | |------|----------|---------| | Coordinate-level sparsity | How many coordinates update? | 66%-90% static, FFN-dominated | | Spectral concentration | Where does update energy live? | Full-rank numerically, low-dimensional dominant | | Principal-space alignment | Relation to source weights' principal directions? | Deliberately avoids them, biased toward low-magnitude coordinates |

This framework generalizes to any post-training method.

Why this matters

1. Fills a theoretical gap: OPD is widely deployed in industry, but its parameter dynamics were almost unstudied. This is the first systematic parameter-space analysis. 2. Counters intuition: "dense supervision = dense updates" is false. Post-training is governed by how the data distribution interacts with the model's current policy, not supervision density. 3. Efficient post-training: with 66%-90% of parameters inactive, you can freeze most weights, design parameter-efficient OPD variants (sparse coordinate updates rather than LoRA-style low-rank), and identify which parameters carry post-trained capabilities. 4. Consistency with related work: Mukherjee et al. (2026a) — RLVR fine-tunes small subnetworks; Zhu et al. (2025) — RLVR learns away from principal directions; Shen et al. (2026), "On the Geometry of On-Policy Distillation" (arXiv:2606.07082) — independently places OPD in a "relaxed off-principal regime" between SFT and RLVR.

Key numbers at a glance

| Number | Meaning | |--------|---------| | 66%-90% | Parameter coordinates with no visible OPD update | | ~1% | OPD update relative norm vs. offline distillation | | ~100% | Performance recovered by training only the sparse subnetwork | | 3 | Analysis dimensions (coordinate sparsity, spectral concentration, principal alignment) | | 2 | Divergent optimizer conclusions: SGD suits RLVR; AdamW suits OPD |

Open questions

1. Why FFN? Do attention/FFN differences hint at FFN storing knowledge while attention controls format? 2. Are sparse coordinates fixed? Do active subnetworks overlap across tasks (math vs. code vs. multimodal)? High overlap would imply a "universal post-training subnetwork." 3. Relation to LoRA? LoRA is low-rank; OPD is sparse-coordinate. Can hybrid sparse-low-rank methods be designed? 4. Scale? Experiments cover 7B-70B; do sparsity patterns hold at frontier scale?

Conclusion

OPD is becoming a standard post-training component, but this paper reminds us: engineering intuition ≠ parameter reality. Dense supervision does not create dense updates — the on-policy data distribution is the dominant force behind parameter sparsity.

For LLM researchers, the methodological lesson: don't just watch loss curves and benchmarks. Occasionally open the parameter black box — which coordinates move, where, and with whom — and you may gain entirely different insights.

References

• Guo Yu, et al. "Dense Supervision, Sparse Updates: On the Sparsity and Geometry of On-Policy Distillation." arXiv:2606.13657, 2026.
• Mukherjee et al. (2026a). "RLVR fine-tunes small subnetworks."
• Zhu et al. (2025). "RLVR learns away from principal directions."
• Shen et al. (2026). "On the Geometry of On-Policy Distillation." arXiv:2606.07082.