When Being the Teacher Is the Problem: Why Your AI Agent Learns to "Procrastinate to Death"
Imagine learning to drive. Your instructor sits in the passenger seat, grading every turn of the wheel and press of the pedal. That's standard on-policy distillation (OPD): the student agent learns on its own trajectories while the teacher scores each token.
Yet a counterintuitive phenomenon has plagued the field: the student drifts off course from a state the teacher can handle—the teacher can even take over from that state and finish the task—yet the student never learns. It keeps stalling until its turns run out.
NVIDIA's Shu Zhao and Tan Yu found the root cause in their new paper *"Capable Teacher, Myopic Scorer"* and proposed a fix called R2S (Rollout-to-Scorer Distillation). Across 12 teacher-student-environment combinations, R2S won every time, outperforming SOTA by an average of 12.8 percentage points.
The Problem Isn't the Student—It's the Teacher's "Myopia"
The mainstream explanation has been compounding error: the student makes small mistakes → enters regions unfamiliar to the teacher → teacher scores become unreliable → drift worsens. Existing methods therefore try to keep students in a "safe zone" via curriculum learning, prefix replay, or teacher intervention.
But this paper uncovered a neglected fact: from the states where students diverge the most, teacher takeovers still complete the task. The teacher's capability is fine; it knows how to reach the goal from there.
The problem lies in the gap between the teacher's *doing* and *scoring*—the rollout-scorer gap:
- When acting: the teacher generates to the end of the episode and sees the full outcome, knowing which moves waste time.
- When scoring: it looks at a single token and assigns a score, blind to what happens next.
- R2S beat three SOTA methods (Guided-OPD, FutureBridge, and another) in all 12 cells
- Average improvement of +12.8 success points
- Even in self-distillation (teacher = student's base model), R2S still works
That's the "myopia." Next-token scoring turns the teacher into a nearsighted judge—one that rewards tokens that *look* safe but actually constitute stalling, since each is fine in isolation but wasteful in the full trajectory.
In outcome-based RL terms: if a trajectory fails, the whole trajectory gets zero reward. Agentic OPD has no terminal penalty, so students are never punished for procrastinating.
A Everyday Analogy
Imagine training an intern two ways:
1. Score every step: each line of code gets a "looks fine." Whether the project ships on time is irrelevant. 2. Judge the outcome: only delivery counts. Detours are the intern's problem.
The first is current agentic OPD; the second is outcome-based RL. Under approach one, the intern learns "safe procrastination"—every step scores well, but the project never finishes, because delay itself is never penalized.
R2S: Fix the Teacher First, Then Teach the Student
R2S's core insight: instead of fixing the student, fix the teacher first.
1. Mine divergence states: run untrained students through episodes; find states of maximal teacher-student disagreement. 2. Teacher takeover: from each state, run K teacher takeovers to test task completion. 3. Reward-free filtering: use consensus or an LLM judge to verify which takeovers actually reached the goal—no external reward signal needed. 4. Repair the scorer: fine-tune the teacher on this "outcome-aware" takeover data. Specifically, from the divergence turn onward, each assistant token's target is replaced with takeover-informed scores. 5. Freeze and distill: the repaired teacher is frozen and serves as the sole scorer for standard OPD training.
Key innovation: R2S doesn't intervene in the student during training (as Guided-OPD does); it repairs the teacher's myopia *before* training. The repaired scorer encodes "this token leads to stalling" into its scores.
Experimental Results
Across three environments (WebShop, WebArena, and a third benchmark), two teacher models, and four student models—12 combinations:
My Take
What I admire most is the diagnostic approach. Most OPD improvements ask "how do we keep the student from drifting?" This paper asks a more fundamental question first: is the teacher actually emitting correct signals?
The answer is no. The teacher's *doing* and *scoring* are two systems—one has full trajectory context, the other sees only the current token. That information asymmetry is the rollout-scorer gap.
This maps directly onto human education: a good teacher isn't necessarily a good grader. A teacher lecturing knows the whole knowledge map and can guide students to the destination; but grading step-by-step, they'd reward answers that are locally correct yet globally off-track.
R2S is also engineering-elegant: no training-loop changes, no student architecture changes—just compute up front (19,200 takeover episodes + one teacher fine-tune) to repair the scorer. Much cleaner than bolting interventions onto the training loop.
One limitation: it requires resettable environments—takeovers must start from the student's actual state, so the environment must support save/load. All three benchmarks support this; real-world deployment may not. For the training stage, though, the requirement is entirely reasonable.
Conceptual Takeaways
Another case of judgment-gate decoupling: the teacher's *doing* (judgment: knowing the path to the goal) and *scoring* (gate: controlling the student's training signal) are independent functions. R2S's contribution is showing the gate is broken, not the judgment.
Another case of scalar illusion: a token-level score is a scalar that hides whether the token causes stalling in the full trajectory. The same high score can mean "genuinely advancing the task" or "safely stalling."
---
Paper: Capable Teacher, Myopic Scorer: Reward-Free Teacher Repair for Agentic On-Policy Distillation Project page: https://tree-shu-zhao.github.io/r2s/ Code: https://github.com/Tree-Shu-Zhao/miles-agent