When AI Agents Fail Step by Step in Training: How Alibaba's RTI Corrects Errors at Rollout Time and Breaks the Capability Ceiling

The Problem: On-policy RL's Capability Ceiling

The standard recipe for RL-training an LLM agent is: sample G trajectories per task, score them with a reward function, and reinforce successes while discouraging failures (e.g., GRPO). The hidden assumption is that the model can complete the task — it just does so with low probability, and RL amplifies that low-probability success path.

In reality, when tasks exceed the model's current capability boundary, all G trajectories fail, rewards are all zero, advantages and gradients vanish, and the model learns nothing. This is the "on-policy reasoning bottleneck." Existing fixes bring in a stronger teacher model (e.g., LUFFY, OPD) to supply success trajectories — but then the student's ceiling is capped by the teacher.

Key Insight: Failure Is One Misstep, Not Inability

Analyzing many failure trajectories, the RTI team found a recurring pattern: agents usually possess the knowledge and tools needed, but make one small early error at a task-critical step, sending subsequent reasoning into a failure loop.

Two examples from the paper:

• τ²-Telecom (customer service): The user asks to check WiFi speed. The agent should have the *user* run run_speed_test, but instead tries to call this user-side tool itself, errors repeatedly, and collapses.
• Navigation: Asked for a route to a specific plaza, the agent searches and picks the first inexact POI match instead of refining the query with a city qualifier.

In both cases, correcting one critical decision would let the rest of the trajectory succeed.

RTI's Three Components

RTI intervenes during rollout when an entire sampling group fails, turning a failed trajectory into a corrected successful one for the policy to learn from. Since corrected trajectories are off-policy, three safeguards are needed:

1. Validator–Corrector–Verifier pipeline (self-intervention)

All three roles are played by the policy model itself:
• Validator: diagnoses whether the current step is a harmful deviation, using reward-side evidence (rubrics, not answers). It is reward-oriented, not process-prescriptive — it tolerates suboptimal-but-recoverable choices.
• Corrector: performs a minimal correction — changing only the critical error while preserving the original reasoning style (a "silent calibrator"), not rewriting with an oracle answer.
• Verifier: checks that the correction matches the gold requirement, leaks no privileged information, and remains a plausible continuation of the policy's behavior. Removing it crashes performance: τ²-Telecom Pass@1 falls from 61.40 to 26.00 — worse than uncorrected GRPO (35.33).

2. Perplexity gating

Corrections may contain tokens the policy would rarely generate, destabilizing training. RTI computes the corrected response's perplexity under the rollout policy and discards it if above threshold δ:
\[PPL_{roll}(\tilde{u}_k) = \exp\left(-\frac{1}{T_k} \sum_{t=1}^{T_k} \log \pi_{\theta}^{roll}(\tilde{u}_k^t | h_{1:k-1}, \tilde{u}_k^{<t})\right)\]

Hyperparameter studies show δ=5 is the sweet spot: δ=0 degenerates to GRPO, δ=∞ skips filtering (high variance), δ=20 is too loose.

3. Reasoning-shift-aware token reweighting

Tokens are weighted by the ratio of their probability under the correction prompt vs. the original rollout prompt:
\[\rho_t = \exp\left(\log \pi_{cal}(\tilde{u}_k^t | \cdots) - \log \pi_{\theta}^{roll}(\tilde{u}_k^t | \cdots)\right)\]

clipped to \([1-\epsilon_{cal}, 1+\epsilon_{cal}]\) and multiplied into the GRPO advantage, so correction-critical tokens get larger learning weight.

Results: Breaking the Ceiling

| Method | Navigation p@1 | τ²-Telecom p@1 | τ²-Retail p@1 | τ²-Airline p@1 | Claw-Eval-Hard20 p@1 | |---|---|---|---|---|---| | Qwen3-8B (base) | 22.67 | 14.17 | 57.02 | 35.53 | 16.00 | | GRPO | 35.33 | 35.33 | 55.48 | 50.22 | 24.38 | | LUFFY (teacher-guided) | 36.00 | 36.00 | 72.59 | 49.34 | 32.81 | | RTI | 57.33 | 61.33 | 81.14 | 55.92 | 50.00 | | Qwen3-235B-A22B (ref) | 28.00 | 16.67 | 64.04 | 47.15 | 32.00 | | GPT-4.1 (ref) | 23.33 | 18.67 | 58.99 | 52.19 | 34.00 |

Qwen3-8B + RTI hits 57.33 on Navigation, surpassing the 235B sibling and GPT-4.1. On Claw-Eval-Hard20 it reaches 50% vs. GRPO's 22.5% — roughly doubled.

Counterintuitive finding: self-correction beats a stronger teacher

Using a 397B model to correct a 9B policy's trajectories performed worse than self-intervention: teacher corrections lie further from the student's distribution, so the PPL gate filters more of them and learning signals are less stable. The best teacher is the one matched to your current level — here, yourself.

Training dynamics

As training progresses, the average PPL of intervened steps declines, and pre-intervention Pass@8 rises above GRPO's sampling success rate — the policy genuinely internalizes corrected behavior rather than depending on intervention.

Engineering Takeaways

1. "One wrong step, then everything wrong" is the typical agent failure mode — failures are often local critical-step errors cascading, not general incapability. 2. Off-policy data is usable with filtering + weighting: Verifier filters leakage, PPL gating filters distribution shift, token reweighting separates critical corrections from boilerplate. 3. Self-intervention philosophy: no teacher model to maintain, no teacher-student distribution mismatch; self-correction ability is trainable — a refined cousin of self-play. 4. Intervention timing: RTI only intervenes when an entire sampling group fails, avoiding over-intervention and dependency while maximizing marginal value.

Code and Paper

• Paper: *Rollout-Time Intervention: Unlocking Agentic Reinforcement Learning Beyond Policy Limits* — Alibaba Token Hub, Alibaba Group / Amap / HKUST (Guangzhou)
• arXiv: 2610.13438
• Code: https://github.com/Alibaba-NLP/qqr (full RTI implementation, Validator/Corrector/Verifier prompt templates, evaluation scripts for five benchmarks)

One-line summary: Agent failures are often not "can't" but "one critical step gone wrong"; correcting that step at sampling time with the model itself works better than hiring a stronger teacher. The capability boundary is not fixed — it is a function of the current training method.