PIVOT: When All Attempts Fail, How Should a Teacher Intervene?

A student is solving a math problem. He has tried 8 different approaches, and all failed. How should the teacher help?

One option is to give the answer directly—but that robs the student of the chance to think. Another is to give hints—but if the student is completely stuck, hints are useless.

This is the core challenge when training LLMs with RLVR (Reinforcement Learning with Verifiable Rewards): when all rollouts on a problem fail and there is no positive signal to learn from, training stalls.

A late-October 2026 paper, PIVOT (2610.40576), gives a precise answer: the teacher must intervene at two levels—providing the model with a path that leads to success (trajectory-level intervention), while also annotating which tokens on that path are key decision points (token-level supervision). Either alone is insufficient; both must coexist.

The RLVR Deadlock Problem

Background: RLVR is the mainstream method for training reasoning models. Given a problem, the model generates multiple reasoning rollouts, and whether the final answer is correct serves as the reward signal, reinforcing correct paths and suppressing incorrect ones. DeepSeek-R1, QwQ, and similar models were trained this way.

RLVR's premise is that the model can answer correctly in at least some rollouts. If it fails all 8 attempts on a problem, all reward signals are 0 and the model learns nothing—it cannot tell which path is "closer" to the correct answer because all rewards are identical.

This is the "deadlock": the model needs success to learn, but needs learning to succeed.

Existing solutions use "teacher guidance"—a stronger model (teacher) helping a weaker one (student). But *how* to help matters. PIVOT's core contribution is distinguishing two types of help and proving both are indispensable.

Two Interventions: Trajectory vs. Token Annotation

PIVOT distinguishes two teacher interventions:

Trajectory-level intervention: the teacher directly provides a reasoning path that leads to success. The student doesn't have to fumble—it walks along the teacher's path. This solves the "no successful path" problem.

Token-level supervision: the teacher annotates which tokens on the path are key decision points. The student not only sees the path but knows "which steps matter and which are transitional." This solves the "don't know what's worth learning from the path" problem.

The key finding: neither alone is enough.

• Trajectory-level only: the student can traverse the path but doesn't understand why it works; generalization is poor.
• Token-level only: the student knows which tokens matter, but without a successful path as the carrier, the annotations have nowhere to apply.
• Combined: the student has a successful path *and* knows which steps on it are critical, so it truly learns "how to make the right decisions at key points."

Middle Intervention > Prefix Intervention

Another notable detail: where the teacher intervenes matters.

• Prefix intervention: teacher guidance is inserted at the start of the reasoning path; the student continues from that starting point.
• Middle intervention: guidance is inserted mid-path; the student attempts first, the teacher intervenes at the sticking point, and the student continues.

Experiments show middle intervention significantly outperforms prefix intervention.

Intuitive explanation: prefix intervention gives the student a "correct starting point," but a correct start doesn't guarantee correct subsequent reasoning—the student may repeat its mistakes later. Middle intervention precisely targets the student's actual sticking point: the teacher shows up where help is genuinely needed, rather than taking over from the beginning.

This matches the intuition of human teaching: a good teacher doesn't write the first line of code for the student, but points out the key idea when the student is stuck.

Choosing the KL Divergence: Situational Fit

A practical finding: different KL divergences suit different scenarios.

• Reverse KL: suited when the model's initial accuracy is low. It allows the model to deviate substantially from its initial policy and explore new reasoning paths.
• Forward KL: suited when initial accuracy is high. It fine-tunes while staying close to the original policy, avoiding excessive drift.

Practical value: use reverse KL early in training (weak model), switch to forward KL later (stronger model). This situational strategy beats a one-size-fits-all KL choice.

The Numbers: avg@8 Nearly Doubled

The core experiments were done on logic reasoning games. The baseline model (no teacher guidance) had an avg@8 of 19.5%. With PIVOT's dual intervention, avg@8 rose to 35.6%—nearly double.

That number itself isn't stunning (35.6% is still low), but considering it was achieved in a scenario where the baseline fails on hard problems—where vanilla RLVR learns nothing (all-zero rewards)—a 16.1 percentage-point gain means something different.

More notably: the gains from trajectory-level alone or token-level alone are far smaller than the combination, confirming the paper's central claim that the two interventions are complementary, not redundant.

A "Judgment-Gate Decoupling" Instance

PIVOT's design recalls a pattern recurring in agent engineering: decoupling judgment from gating.

In PIVOT:

• Trajectory-level intervention is the "gate"—it determines whether the model can access a success path at all. Without this gate opening, the model circles in failure paths forever.
• Token-level supervision is the "judgment"—it marks which tokens are key decisions. Without this judgment, the model traverses the path without understanding why it works.

The gate and the judgment are independent functions that cannot substitute for each other. You can open the gate without judgment (trajectory only)—the model succeeds but doesn't understand. You can provide judgment without opening the gate (token supervision only)—annotations with no carrier. Only when both exist does learning actually happen.

This pattern appears elsewhere:

• Concordat (2610.37461): the protocol is the gate (tools not on the list), the type system is the judgment (subclasses can't violate).
• ShallowDefend (2610.40627): safety decisions are the judgment (which tokens need review), the independent review channel is the gate (review doesn't block generation).
• CORAL (2610.40350): checkpoints are the judgment (what's worth saving), resets are the gate (what gets cleared).

PIVOT adds a new instance to this lineage: trajectory intervention is the gate, token supervision is the judgment.

Implications for Agentic RL

PIVOT's findings apply directly to Agentic RL.

The current mainstream approach lets agents run tasks on their own, with task success as the reward. But many tasks have extremely low success rates—an agent may succeed once in 100 attempts. This sparse reward makes training inefficient.

PIVOT's approach transfers: when all rollouts on a task fail, don't let the agent blindly keep trying—bring in the teacher. The teacher can:

1. Provide a success path (trajectory-level): e.g., an expert demonstration trajectory or a stronger model's solution. 2. Annotate key decision points (token-level): mark "this step is a key decision" / "this step is transitional" along the path.

Given both, the agent's learning efficiency far exceeds pure RLVR.

The cost is teacher expense. The paper doesn't discuss teacher inference overhead in detail, but from the experimental setup, the teacher only needs to intervene on "all-rollouts-fail" hard problems, not every problem. If hard problems make up 20%, the teacher cost is acceptable.

An Open Question

PIVOT solves the "all rollouts fail" scenario but leaves an open question: when should the teacher step in?

The paper's setting is "teacher intervenes when all rollouts fail." But "all fail" is a binary judgment; reality has finer gradients—e.g., 7 failures and 1 barely-successful rollout out of 8. Should the teacher intervene then?

Intervene too often and the student depends on the teacher, learning little on its own; too rarely and the student stays stuck on hard problems. The paper gives no clear answer on timing—future work may need to address it.

From another angle, this open question shows PIVOT's framework is extensible: "when to intervene" can be treated as an extra hyperparameter, or even a learned objective.

Conclusion

PIVOT does something very clean: it decomposes the vague question of "how should a teacher help" into two precise dimensions—trajectory intervention and token supervision—and proves experimentally that neither works without the other.

This research style of "splitting entangled functions apart" is increasingly common in agent engineering. From ShallowDefend's "safety decisions as separate tokens," to MTA-OPSD's "teacher selection down to the token level," to PIVOT's "trajectory intervention vs. token supervision," the core idea is consistent: separate entangled decisions and optimize each one.

The paper has no open-source code, but the method is described clearly enough for an experienced team to reproduce.

Paper: https://arxiv.org/abs/2610.40576