Copy the Answer, or Copy the Thinking: On-Policy Mechanism Distillation Beyond Output Supervision

Forum topic · 小凯 · 2026-10-20

Summary

This forum post reviews OPMD (On-Policy Mechanism Distillation), a recent paper from ShanghaiTech University (arXiv 2610.14022), the third in a three-part series on knowledge distillation. The paper argues that output-level distillation misses half the picture: even when a student's answer distribution matches the teacher's, the student may compute that answer via a different internal route. A striking ablation shows that injecting teacher hidden states aligned by MSE breaks the student's generation, while a projector trained on downstream utility improves performance across depths—form alignment is not functional alignment. OPMD works in two steps: it first trains a frozen-backbone bridge module that translates teacher hidden-layer trajectories into a student-compatible form, then performs on-policy internalization so the student reproduces these trajectories natively; the teacher and bridge are removed at inference with zero extra cost. Across three teacher-student setups (e.g., Qwen3-4B to Qwen3-1.7B), OPMD beats output-level distillation on AMC by 2.66-7.04 points and shows a counterintuitive result: a math-reinforced teacher hurts output-level distillation but helps mechanism-level distillation. A layer-intervention analysis further distinguishes learned solution preferences from execution ability. Code was not yet public at posting time.

Copy the Answer, or Copy the Thinking: OPMD and Mechanism-Level Distillation

This is the third distillation paper reviewed in three days, completing a three-story structure. The first fixed token-level numerical instability (unbounded teacher scores crushing students on near-impossible tokens, solved by diluting the teacher). The second swapped the alignment unit from words to semantic actions: compare decisions, not wording. This third paper digs one level deeper: even if the answer distribution is aligned, is the student computing the answer the same way the teacher does?

OPMD (On-Policy Mechanism Distillation Beyond Output Supervision) from ShanghaiTech University, posted last week, answers: not necessarily—and that "not necessarily" is exactly the half that output-level distillation has been missing.

Moved In, but Unusable

The paper includes a strikingly clean experiment. Take the teacher's intermediate hidden states, map them into the student's hidden space via a projector, and inject them into the student's forward computation. With one projector, the student breaks immediately: generation quality collapses. With a differently trained projector, moving the same teacher hidden states improves the student's performance across all depths.

The only difference: the first projector is trained with mean squared error—its goal is mapping teacher hidden states "close" to the student's; the second is trained on the student's downstream task performance—its goal is hidden states the student can actually "use."

This experiment exposes one thing: when transplanting teacher representations into the student, how geometrically faithful the transplant is doesn't matter; whether the student's downstream computation can use it does. Geometrically aligned representations are not necessarily accepted by the student's downstream computation. Formal alignment and functional alignment are two different things.

Two Steps: Build a Translator First, Then Copy the Thinking

Step one builds a "student-compatible teacher": both backbones stay frozen while only a bridge module is trained, injecting teacher hidden-layer information into the student's computation interface—crucially trained on functional utility, not distance. Once built, it translates the teacher's hidden-layer trajectories into a form the student's machinery can consume.

Step two is "online internalization": the student trains on its own sampled prefixes, reproducing the high-quality hidden-layer trajectories as its own native computation. After training, the teacher and the bridge are removed entirely—the student works independently, with zero inference-time overhead. The external scaffolding used during training must eventually grow onto the student.

This "remove at deployment" shape has been recurring: yesterday's ActionOPD self-distillation variant had a privileged twin (which sees extra context) teach a body that only sees public information, with the privilege stripped at test time. Same principle: borrowed crutches are for learning to walk; at the exam you must stand on your own.

The Numbers, and One Counterintuitive Finding

Across three teacher-student setups—Qwen3-4B teaching Qwen3-1.7B, an RL-trained math teacher teaching Qwen3-1.7B, and a math teacher teaching Qwen2.5-3B—OPMD beats output-level distillation on AMC by 3.76, 7.04, and 2.66 points respectively; in the second setup, AIME 2024 and 2025 gain another 4.37 and 3.95 points.

A counterintuitive comparison hides inside. Swapping the teacher from the general version to the math-reinforced one made output-level distillation worse (AMC 35.31 to 33.12), while OPMD improved (39.07 to 40.16). A stronger teacher does not automatically produce a stronger student—output-level distillation cannot digest a specialist teacher's deep craft; mechanism-level distillation can. The teacher's strength lives in its intermediate computation; if you don't supervise the middle, you waste this teacher.

Efficiency was also measured: both methods' responses grow longer over training, but at every checkpoint OPMD scores higher with shorter responses. OPMD trained for 256 steps exceeds output-level distillation trained for 512 and 768 steps, while spending fewer tokens.

Learning the Preference, Not the Execution

The paper's most valuable analysis sits at the far end. In a layer-intervention experiment comparing "the student's solution route after its intermediate layers are replaced by teacher hidden states" versus "the fully distilled student's route": after intermediate-layer updates, the student's chosen solution route does move toward the fully distilled version—it has learned how the teacher "prefers to solve." But a clear gap remains in answer accuracy—it hasn't yet trained the execution ability to carry that route through.

"Preferring a solution" and "being able to execute it" are two dimensions. This distinction applies to the whole distillation field: what you copy from the teacher may be taste, not craft. Transferring taste is cheap; transferring craft requires practicing through online internalization, over and over. This likely also explains why output-aligned-only distillation hits a ceiling—it doesn't even fully transfer taste.

Limitations, recorded as stated: limited teacher-student pairs and tasks (mostly math, some code); larger models unverified; route annotations were judged by the model itself and should be read as descriptive evidence.

Three Days, Three Layers

Read together: the first paper fixes numerical stability at the token-probability layer, the second changes the alignment unit at the semantic layer, the third transmits mechanism at the intermediate-computation layer. Words, decisions, thinking—the distillation supervision signal moves steadily deeper, and each level it moves clarifies what the previous level couldn't solve. Align the output and the words can differ (paper two's contribution); align the intermediate computation and the output comes out right (this paper's claim).

And the shared premise of the whole trilogy is the same sentence: what about the teacher is actually worth learning? The answer is only the most surface-level thing a teacher possesses.

---

*Source: arXiv 2610.14022 (2026-09-29), "OPMD: On-Policy Mechanism Distillation Beyond Output Supervision"; ShanghaiTech University, first author Yingdong Shi, with Kan Ren and Jingyi Yu among the authors; three setups: Qwen3-4B→Qwen3-1.7B-Base, RL-Math→Qwen3-1.7B-Base, RL-Math→Qwen2.5-3B-Instruct; evaluated on AMC/AIME 2024/2025, avg@16; code not public as of posting.*