RL Training Sudden Collapse: The Model Didn't Get Dumber—It Forgot How to Speak the Format

Forum topic · ✨步子哥 · 2026-06-25

Summary

This post analyzes a 2026 paper by Yupu Hao et al. (ISCAS & CASIA) on why multi-step tool-use reinforcement learning collapses mid-training. The paper shows the collapse is not reward hacking or capability forgetting but structural output collapse: probabilities of control tokens like <tool_call> spike toward 1.0, breaking the XML formatting that parsers rely on. A format-repair experiment proves the model retains full capability—after reformatting outputs, success rate recovers from 5% to 72%. The authors test five supervisory signals (SFT-then-RL, off-policy, hint-based, erroneous, and interleaved SFT-RL). All prevent collapse; interleaved SFT-RL performs best in-distribution (12-18 points above pure RL across 7 benchmarks) but is less robust under out-of-distribution evaluation. Learning rate also matters: 1e-5 collapses over 80% of the time, while 2e-6 is stable but 5x slower. Practical takeaways: collapse is inherent to pure RL, not a bug; any SFT-style supervision stabilizes it; OOD evaluation is essential; and output format should be treated as part of model capability, not mere packaging.

A Sudden Cliff in the Training Curve

You're training a tool-use agent. The first 200 steps go smoothly: the model learns to call search, query databases, and compose answers, with success rate climbing steadily from 30% to 75%. You think 100 more steps will suffice for convergence.

At step 217, success rate drops to 5%.

Not a gradual decline—a cliff-edge collapse. And the failure mode is strange: the model hasn't gotten dumber. Its reasoning remains sound, its tool choices remain correct. But its output format breaks: garbled text where <tool_call> should appear, unclosed XML tags, missing parameter names. The parser sees malformed output and judges everything wrong.

Restart training, change the seed: collapse at step 189. Change the learning rate: collapse at step 251. Larger batch size: collapse at step 163.

This is not an isolated case. A June 2026 paper by Yupu Hao et al. (Institute of Software, CAS and Institute of Automation, CAS), *Why Multi-Step Tool-Use Reinforcement Learning Collapses and How Supervisory Signals Fix It*, systematically documents this phenomenon and identifies its root cause.

Not a Capability Problem—a Format Problem

The researchers ran a key diagnostic. After collapse, they performed a "format repair" on model outputs: another model reformatted them, preserving the original reasoning and tool choices while only fixing XML tags and parameter structure.

Result: after format repair, success rate recovered from 5% to 72%.

The model's capability was never lost. It still knew which tools to call, what parameters to pass, how to reason. It simply couldn't "say it out loud"—the output format broke, and the format failure masked the capability.

This is a counterintuitive finding. RL collapse is usually attributed to reward hacking or capability forgetting. But this paper reveals a new collapse mode: structural output collapse. The model didn't forget how to use tools—it forgot how to say "I want to use a tool."

Probability Spikes on Control Tokens

The root cause hides in the token probability distribution.

Tool-use models learn during SFT to use special control tokens as a protocol with the parser—<tool_call>, </tool_call>, <parameter>, etc.

The researchers found an anomaly in token probabilities around collapse: the log-probabilities of certain control tokens spike several steps before collapse. Normally <tool_call> sits around 0.9 at positions where it belongs. Before collapse, it gets pushed to 0.9999+, squeezing all other candidate tokens to nearly zero.

That sounds like healthy confidence. But RL gradient updates amplify the advantage of high-probability tokens. When a control token's probability is near 1, every gradient update is dominated by that spike—the model learns not "when to use tools" but "output <tool_call> no matter what." Formatting breaks: the token appears where it shouldn't, tags go unclosed, and the rhythm of structured generation is destroyed.

Analogy: imagine a conductor where one beat's gesture is amplified to deafening volume. The musicians can't hear the other beats—the whole piece falls apart. The players didn't forget how to play; the rhythmic signal broke.

Five Supervisory Signals

The paper tested five forms of supervision inserted into RL training:

1. SFT-then-RL: SFT on high-quality trajectories first, then switch to RL. 2. Off-policy supervision: periodic SFT updates using high-quality trajectories from earlier policies—"reviewing while training." 3. Hint-based supervision: partial hints in RL prompts to guide exploration. 4. Erroneous supervision: showing the model wrong trajectories as negative examples. 5. Interleaved SFT-RL: alternating RL and SFT batches.

Which Works?

Results come in two layers:

Collapse prevention: all five signals prevent collapse. Any form of SFT supervision keeps control token probabilities from spiking and keeps formatting stable. Collapse isn't inevitable for RL in general—it's inevitable for *pure* RL.

Performance: Interleaved SFT-RL performs best on most benchmarks. It retains RL's exploration ability while periodically refreshing format via SFT. Across 7 tool-use benchmarks, Interleaved beats pure RL by 12-18 percentage points on average.

But there's a trade-off: Interleaved SFT-RL is worse under OOD (out-of-distribution) evaluation. When test-time tool formats or task structures differ slightly from training, pure RL models are more robust—they learned "how to explore," not "how to remember a format." This is a fundamental tension: stability vs. generalization. Interleaved is currently the best compromise, not the endgame.

The Learning Rate Effect

LR 1e-5: collapse probability 80%+
LR 5e-6: collapse probability 30%
LR 2e-6: almost never collapses, but 5x slower training

Large learning rates push probabilities to extremes in single updates, making spikes easier to trigger. The paper's conclusion: learning rate is not a hyperparameter tuning issue—it's a stability issue. Choose LR by "doesn't collapse," not "converges fast."

What This Means

For anyone doing agentic RL:

Collapse is not your bug; it's a property of RL. Don't blame your code, data, or reward design—first check the control token probability distribution. Spikes indicate this phenomenon.
Pure RL is not enough; you need supervision. All five signals work. With high-quality trajectories, Interleaved SFT-RL is the safest choice; otherwise off-policy supervision works.
OOD evaluation is mandatory. Interleaved looks dominant in-distribution, but OOD reveals its limits. The paper stresses: agentic RL papers reporting only in-distribution results are untrustworthy.
Format is part of capability. The deepest insight: the model's capability is intact; what breaks is the format that expresses it. In structured-output settings, format isn't packaging—it's the capability itself.

A Broader Metaphor

Are AI "capability" and "expression" the same thing? Conventionally, capability is internal and expression external. This paper shows that in structured-output settings, they are coupled: when format collapses, capability becomes unusable—externally, the model just looks dumber.

There's a parallel with human cognition: aphasia patients know what they want to say but cannot say it. RL collapse is a kind of "aphasia"—the model didn't forget how to reason, only how to say "I want to reason."

SFT supervision periodically reminds it how to speak. The implication: future RL training may need to treat expressive stability as a first-class citizen, rather than fixing collapse after it happens. The Interleaved scheme is a first step, not the last.

---

Paper: Why Multi-Step Tool-Use Reinforcement Learning Collapses and How Supervisory Signals Fix It Code: https://github.com/hypasd-art/Tool-RL-Box Authors: Yupu Hao, Zhuoran Jin, Huanxuan Liao, Kang Liu, Jun Zhao (Institute of Software, CAS & Institute of Automation, CAS)