An Overlooked Detail
In knowledge distillation for large language models, a teacher model provides a target distribution for every token, and the student model fits it. The usual practice: pick one teacher and let it guide the entire response from start to finish.
But a team from TU Munich, Kyoto University, and Aalto University asks a counterintuitive question in a new paper: within the same response, do tokens at different positions really suit the same teacher?
The answer: no. And the variation is far bigger than you might expect.
Pilot Experiment: 50% of Tokens Want a Different Teacher
The paper's pilot experiment prepared multiple candidate teacher models (constructed via different degrees of "reference-answer exposure") and measured, at each token position within a response, which teacher's guidance was most useful.
Result: within a single response, the overlap rate of the "best teacher" across token positions is only about 50%.
To put that in perspective, consider a math reasoning chain: "Let x be..., substitute into the equation..., solve for x=...". Tokens defining the variable at the start may suit a teacher strong in structured reasoning; tokens doing substitution in the middle may suit one strong in numerical computation; final answer tokens may suit one good at formatting. Sticking to one teacher means roughly half the tokens are "making do."
A further finding: the best teacher is usually the one most different from the student model (largest KL distance). This matches intuition — a teacher must be "far" from the student to provide useful signal, but the direction of "far" varies by token.
MTA-OPSD: Let Each Token Choose Its Own Teacher
Based on this finding, the paper proposes Multi-Teacher Adaptive On-Policy Self-Distillation (MTA-OPSD), the first token-level adaptive distillation method.
The core mechanism has two steps:
1. Build a Multi-Teacher Pool
Instead of one teacher, prepare a pool of candidates. Each teacher is constructed with a different degree of "reference-answer exposure" — some see the full reference answer, some only part, some none. This produces teacher signals of different styles.
2. Adaptive Selection per Token
For each token, the student picks a "best teacher" from the pool, via one of two mechanisms:
- Majority Voting: multiple candidate teachers vote, and the most-endorsed one is selected
- KL-based Selection: pick the teacher most distant from the student model — because "far" teachers carry the most information
- MTA-OPSD improves over baseline models by +5.9 points on average
- It improves on all existing OPSD (On-Policy Self-Distillation) methods — not a lucky win in one configuration, but a systematic improvement
- Ablations confirm: token-level selection > response-level selection > fixed teacher
- ShallowDefend (2610.40627): splits the "safety judgment" out of response generation into an independent decision token
- Concordat (2610.37461): moves safety rules from the prompt to the interface layer, using type systems as gates
- MTA-OPSD: splits "teacher selection" from response level down to token level
The key to this design: selection is dynamic and token-level. The first token of a response may pick teacher A, the second switch to teacher B, the third back to teacher A.
Results: A Consistent +5.9 Point Gain
The paper evaluates MTA-OPSD on Qwen3 models, on math reasoning.
Key numbers:
A notable detail: majority voting and KL-based selection perform similarly, and combining them is no better than either alone. This suggests they capture the same signal (which teacher is most useful for this token), just measured differently.
Why This Finding Matters
The paper's core contribution is not the MTA-OPSD method itself, but a neglected fact it reveals: teacher preference is token-level, not response-level.
This challenges an implicit assumption in current distillation research: one response corresponds to one best teacher. That assumption underlies two mainstream practices:
1. Fixed teacher: the same teacher for the entire training process (full reference-answer exposure) 2. Response-level adaptation: different teachers per response, but constant within a response
MTA-OPSD shows both are approximations at insufficient granularity. The true optimum lies at the token level.
An Analogy: From "One-on-One Tutoring" to "Going Classroom to Classroom"
Traditional distillation is like one-on-one tutoring: one teacher teaches start to finish, regardless of whether a topic suits that teacher's style.
Response-level adaptation is like "switching teachers per subject": a math teacher for math, a language teacher for language. But within one math class, the algebra part and the geometry part may suit different teaching styles.
MTA-OPSD is like a flexible class system: every knowledge point (token) goes to the best-suited teacher, then returns. The cost is higher scheduling complexity; the benefit is higher learning efficiency.
Echoing the "Judge-Gate Decoupling" Trend
This paper connects to an emerging cross-paper theme. Multiple 2026 papers are doing the same thing: splitting a seemingly indivisible decision into finer granularity.
Limitations and an Honest Assessment
The paper has notable limitations:
1. Only math reasoning tested: whether token-level teacher preference holds for code generation or general writing is untested. Math reasoning has strong sequential structure, which may amplify teacher differences 2. Teacher pool depends on reference answers: the multi-teacher construction requires reference answers; how to build a pool without them is not discussed 3. Computational overhead: computing and selecting among multiple teachers per token costs more than a fixed teacher; the paper does not report the overhead ratio in detail
But the core finding — teacher preference is token-level — is an empirical result that stands regardless of the method. Even if MTA-OPSD itself is not optimal, the finding changes our understanding of distillation.
---
Paper: https://arxiv.org/abs/2610.40574
HTML version: https://arxiv.org/html/2610.40574v1
Open-source code: https://github.com/xingzhejun/MTA-OPSD