Why RL Training Collapses Midway: The Model Forgets How to Speak, Not How to Think

Forum topic · ✨步子哥 · 2026-06-25

Summary

A 2026 paper from researchers at the Chinese Academy of Sciences (Yupu Hao et al.) systematically documents a failure mode in multi-step tool-use reinforcement learning: success rates suddenly plummet mid-training even though the model's reasoning and tool choices remain correct. Diagnosis experiments show the collapse is caused by broken structured output formatting—malformed XML tags and missing parameters that parsers reject. After format repair, performance recovers from 5% to 72%, proving capability is masked, not lost. The root cause is probability spikes on control tokens like <tool_call>, which pure RL gradients amplify until generation structure breaks down. The authors test five supervisory signals; all prevent collapse, and Interleaved SFT-RL performs best in-distribution (12-18 points above pure RL), though pure RL remains more robust out-of-distribution. Learning rate is also critical: 1e-5 collapses over 80% of the time while 2e-6 rarely collapses. The paper argues format stability must be treated as a first-class concern in agentic RL training.

A Training Curve That Suddenly Falls Off a Cliff

You're training a tool-use agent. The first 200 steps go well: the model learns to call search, query databases, and compose answers, with success rate climbing steadily from 30% to 75%. You think another 100 steps will converge it.

At step 217, success rate drops to 5%.

Not a gradual decline—a cliff-edge collapse. And the failure mode is strange: the model isn't getting dumber. Its reasoning remains coherent and its tool choices remain correct. But its output format breaks: where it should emit <tool_call>, it emits gibberish; XML tags go unclosed; parameter names disappear. The parser sees malformed output and marks everything wrong.

Restart with a different seed: collapse at step 189. Different learning rate: collapse at step 251. Bigger batch size: collapse at step 163.

Every time: climb halfway, then fall off suddenly.

This isn't an isolated case. Yupu Hao et al. (Institute of Software and Institute of Automation, Chinese Academy of Sciences), in a paper published June 2026—*Why Multi-Step Tool-Use Reinforcement Learning Collapses and How Supervisory Signals Fix It*—systematically documented this phenomenon and found its root cause.

Not a Capability Problem, a Format Problem

The researchers ran a key diagnostic. After collapse, they applied a "format repair" to the model's outputs—using another model to reformat the outputs while preserving the original model's reasoning content and tool choices, fixing only XML tags and parameter structure.

Result: after format repair, success rate recovered from 5% to 72%.

This means the capability wasn't lost. The model still knew which tools to call, what parameters to pass, how to reason. It simply "couldn't say it"—the output format broke, and capability was masked by the formatting failure.

This is a counterintuitive finding. RL training collapse is usually attributed to reward hacking or capability forgetting. But this paper reveals a new collapse mode: structured output collapse. The model didn't forget how to use tools—it forgot how to say "I want to use a tool."

Probability Spikes on Control Tokens

The root cause hides in token probability distributions.

Tool-use models learn special control tokens during SFT to mark structured output—e.g., <tool_call> starts a tool call, </tool_call> ends it, <parameter> marks parameters. These tokens are the "protocol" between model and parser.

Examining token probability distributions before and after collapse, the researchers found an anomaly: the log probabilities of certain control tokens started spiking several steps before collapse.

Normally, <tool_call>'s probability at its expected position is around 0.9—the model is confident a tool call starts here. But before collapse, this probability was pushed to 0.9999+, with all other candidate tokens squeezed toward zero.

This sounds like "the model is more confident," which should be good. The problem: RL gradient updates amplify the advantage of high-probability tokens. Once a control token's probability approaches 1, any gradient update is dominated by that spike—the model learns not "when to use tools" but "emit <tool_call> no matter what."

So the format breaks. The model emits <tool_call> where it shouldn't, or fails to close tags, because the probability spike disrupts the rhythm of structured generation.

Analogy: imagine an orchestra conductor who normally gives a gesture on every beat. If one beat's gesture is amplified to deafening volume, the musicians are distracted by the roar and can't hear the other beats' cues—the whole piece falls apart. The players didn't forget how to play; the rhythmic signal broke.

Five Supervisory Signals

Having found the root cause, the next step was finding fixes. The paper tested five "supervisory signals"—inserting different forms of supervision at different stages of RL training—to see which prevents collapse:

1. SFT-then-RL: first SFT on high-quality trajectories to teach basic tool-use format, then switch to RL. The classic paradigm. 2. Off-policy supervision: during RL, periodically run SFT updates on high-quality trajectories generated by older policies—"review while training." 3. Hint-based supervision: add hints containing part of the correct answer to RL prompts, guiding exploration in the right direction. 4. Erroneous supervision: show the model wrong trajectories so it learns "don't do this"—negative-sample learning. 5. Interleaved SFT-RL: alternate RL and SFT—one batch RL, one batch SFT, repeating.

Which Works?

Results come in two layers:

Preventing collapse: all five supervisory signals prevent collapse. As long as any form of SFT supervision is mixed into training, control token probabilities don't spiral upward and formats stay stable. This shows collapse isn't inevitable for RL itself—it's inevitable for *pure* RL. Any supervision holds it back.

Performance: Interleaved SFT-RL performs best on most benchmarks. It preserves RL's exploration ability (finding new strategies absent from SFT data) while periodic SFT prevents format collapse. Across 7 tool-use benchmarks, Interleaved beats pure RL by an average of 12–18 percentage points.

But there's a trade-off: Interleaved SFT-RL underperforms pure RL on OOD (out-of-distribution) evaluation. When test-time tool formats or task structures differ slightly from training, pure RL models are more robust—because they learned "how to explore," not "remember a particular format."

This is a fundamental tension: stability vs. generalization. SFT gives stability but limits exploration; RL gives exploration but risks collapse. Interleaved is the best current compromise, not the endgame.

The Learning Rate Effect

The paper also has a practical finding: learning rate affects collapse more than expected.

Learning rate 1e-5: collapse probability 80%+
Learning rate 5e-6: collapse probability 30%
Learning rate 2e-6: almost never collapses, but 5x slower training

This confirms the physical mechanism is indeed "probability spikes"—large learning rates push probabilities more extreme per update, making spikes easier to trigger. A small learning rate gives probability a buffer zone where spikes can't easily form.

The cost of a small learning rate is training time. The paper's conclusion: learning rate is not a hyperparameter tuning problem, it's a stability problem. The criterion for choosing it should be "doesn't collapse," not "converges fast."

What This Means

Direct implications for anyone doing agentic RL:

First, collapse is not your bug—it's a property of RL. If tool-use RL collapses suddenly, don't suspect your code, data, or reward design—first inspect control token probability distributions. If you see spikes, this is the phenomenon the paper describes.

Second, pure RL isn't enough—you need supervision. All five tested signals work; choose based on your scenario. With high-quality trajectory data, Interleaved SFT-RL is currently the most stable choice. Without it, off-policy supervision (good trajectories from older policies) also works.

Third, OOD evaluation is mandatory. If you only look at in-distribution performance, Interleaved looks like it crushes pure RL across the board. But OOD evaluation exposes its limits. The paper emphasizes: agentic RL papers reporting only in-distribution results are not trustworthy.

Fourth, format is part of capability. The paper's deepest insight isn't a specific method but this observation: the model's capability wasn't lost—what was lost was the "format for expressing it." In structured-output settings, format isn't "packaging"; it's capability itself. When format breaks, no amount of capability gets through.

A Broader Metaphor

The paper raises a broader question: are an AI's "capability" and "expression" two separate things?

The traditional view says yes—capability is internal, expression external, capability transmitted through expression. But this paper shows that at least in structured-output settings, capability and expression are coupled. When format collapses, capability becomes unusable—from the outside, the model just looks dumber.

There's an analogy to human cognition. Aphasia patients know what they want to say but can't say it—from outside, they appear to have "lost language ability." But the internal linguistic capability remains; the expression channel is broken.

RL training collapse is, in a sense, the model's aphasia. The model didn't forget how to reason—it forgot how to say "I want to reason."

SFT supervision's role is periodically helping it "review how to speak"—even while learning new reasoning strategies via RL, it must not forget the format for expressing them.

The metaphor isn't perfect, but it points to an important direction: future RL training may need to treat "expression stability" as a first-class citizen, rather than fixing collapse after it happens. The paper's Interleaved approach is a first step in that direction, but not the last.

---

Paper: Why Multi-Step Tool-Use Reinforcement Learning Collapses and How Supervisory Signals Fix It Code: https://github.com/hypasd-art/Tool-RL-Box Authors: Yupu Hao, Zhuoran Jin, Huanxuan Liao, Kang Liu, Jun Zhao (Institute of Software & Institute of Automation, Chinese Academy of Sciences)