RAGEN-2: Template Collapse — When AI Agents Master the Art of 'Correct Nonsense'

Forum topic · 小凯 · 2026-06-20

Summary

Researchers from Stanford, Northwestern, UIUC, and collaborators (advised by Fei-Fei Li and Yejin Choi) introduce RAGEN-2, which identifies a hidden failure mode in multi-turn agent reinforcement learning called template collapse. Models can maintain high or stable entropy—traditionally treated as a sign of healthy, diverse reasoning—while actually producing fluent, well-structured outputs that are largely independent of the input prompt. The paper shows entropy fails to predict final performance (Spearman correlation -0.11 to -0.14), whereas mutual information between input and reasoning traces correlates positively with performance (+0.39) and serves as an early-warning signal. Mechanistically, the authors decompose gradient updates via a signal-to-noise ratio (SNR) framework, showing that input-agnostic regularization (KL and entropy penalties) dominates when prompt-level reward variance collapses, pushing models toward templates. They propose SNR-Aware Filtering—keeping only the top-ρ prompts by reward variance per iteration—which is lightweight, pluggable into PPO/GRPO/REINFORCE++, and consistently improves task performance and input dependence across planning, math, web navigation, code, and tool-use environments.

RAGEN-2 (Wang et al., arXiv:2604.06268, 2026), from teams advised by Fei-Fei Li and Yejin Choi at Northwestern, UIUC, Imperial, Oxford, UW, Microsoft, and Stanford, uncovers a subtle crisis in multi-turn agent RL: template collapse—LLMs quietly learn a "universal template" whose answers look rich and varied but are largely unrelated to the input. Existing metrics miss it; only mutual information catches it.

1. The Illusion: High Entropy ≠ Real Reasoning

Practitioners treat entropy as an iron law: high entropy = diverse outputs = healthy training; low entropy = mode collapse. RAGEN-2 punctures this: a model can have stable or high entropy while completely stopping thinking. Its responses look structured and fluent but barely relate to the input—the model is reciting templates, not reasoning.

Example: given two entirely different math problems (solving x² + 3x + 2 = 0 vs. proving infinitely many primes), the model produces the same generic reply: "First, I need to understand the core requirement... by analyzing the given conditions, the key constraint is... based on this, I construct a reasonable strategy... therefore the answer is..." It fits any problem, contains no actual derivation, yet passes every surface-level check.

2. Why Existing Metrics Go Blind

Entropy only measures "within", not "across". Entropy H(Z|X) = -Σ P(z|x) log P(z|x) measures output diversity for the *same* input. A collapsed model varies wording within one input (entropy looks fine) but produces near-identical structures across inputs.

Rewards are unreliable. Multi-turn agent rewards are sparse. A template that luckily succeeds 50% of the time shows "medium" reward; early templated mistakes can be erased by later correction, yielding positive reward despite broken reasoning.

Length and format don't help either. Reasoning length monotonically decreases under collapse, but too slowly to be a useful signal. High format correctness (valid JSON, complete structure) does not guarantee input dependence at all.

3. Mutual Information Catches It

RAGEN-2 decomposes reasoning quality into two orthogonal dimensions:

1. Conditional entropy H(Z|X): diversity for the same input 2. Mutual information I(X;Z): dependence between input and reasoning—whether different inputs get different reasoning

| | High I(X;Z) | Low I(X;Z) | |--|-----------|-----------| | High H(Z|X) | Ideal: diverse and input-sensitive | Template collapse: diverse but input-independent | | Low H(Z|X) | Input-sensitive but fixed | Total collapse |

Template collapse = high conditional entropy + low mutual information.

Since computing true I(X;Z) online is infeasible, the paper proposes lightweight proxies: for each batch, compare each reasoning chain's log-probability under its own input (matched) vs. other inputs (marginal). MI proxy ≈ matched − marginal; under collapse, matched ≈ marginal and MI → 0. Variants include Retrieval-Acc (discrete retrieval match rate) and MI-ZScore-EMA (continuous estimate with z-score normalization and EMA smoothing).

Key empirical result:

MI vs. final performance: Spearman +0.39 (trajectory-level MI-ZScore)
Entropy vs. final performance: Spearman −0.11 to −0.14

Entropy doesn't just fail to predict performance—its trend points in the *opposite direction*.

4. SNR Mechanism: Why Collapse Happens

Each RL update decomposes as g_total = g_signal + g_task-noise + g_reg:

g_signal: reward differences across trajectories within a prompt (uncontrollable)
g_task-noise: sampling noise + environment stochasticity (uncontrollable)
g_reg: KL + entropy regularization—the same shrinkage on every chain, regardless of prompt (controllable via λ)

When per-prompt reward variance is high, signal dominates and the model learns input distinctions. When variance collapses (increasingly common as training progresses), regularization still pushes updates in an input-agnostic direction, erasing cross-input differences. With SNR(x) = ||g_signal(x)|| / (||g_task-noise(x)|| + ||g_reg||), low SNR → template collapse.

5. SNR-Aware Filtering: The Fix

Each training iteration: 1. Sample G trajectories per prompt 2. Compute reward variance Var(R|X) 3. Rank prompts by variance; keep only top-ρ 4. Update policy only on kept prompts

It requires no extra models or rollouts, and plugs into PPO, GRPO, or REINFORCE++. Reward variance correlates +0.63 with task reward but only −0.14 with entropy and +0.12 with length—an independent signal dimension. Tuning KL/entropy controls *noise*; filtering *amplifies signal*; they are complementary.

Validated on Sokoban planning, MetaMath-style math, web navigation, code execution, and tool use, filtering consistently improves performance and MI. Notably, MI drops earlier than task performance, making it a genuine early-warning indicator, and the filter automatically concentrates gradients on the shrinking set of informative prompts.

6. Behavioral Signatures

Reasoning length declines monotonically across 8 environments—not conciseness, just tighter templates
100% format-valid outputs can be entirely templated
In multi-turn tasks, templated early mistakes can be erased later, so users never notice the agent "gave up thinking" in round one

7. Limitations

1. SNR decomposition assumes signal/noise separability 2. Single-agent only; propagation in multi-agent settings unknown 3. Models might game the filter by inflating reward variance 4. Variance is unreliable under extremely noisy sparse rewards 5. Aggressive filtering narrows exploration; keep rate needs per-task tuning

8. Why It Matters

Conceptually, RAGEN-2 defines a failure mode invisible to all existing metrics—we were monitoring surface diversity, not stability. Diagnostically, swapping entropy for mutual information shifts the question from "is it diverse within one input?" to "is it distinguishable across inputs?" Mechanistically, the SNR account yields testable predictions and a targeted intervention. Engineering-wise, the fix is lightweight and pluggable.

One-line takeaway: LLM agents in RL learn "correct nonsense"—fluent, structured, input-independent. Entropy can't see it, mutual information can; reward-variance collapse is the root cause, and filtering high-variance prompts fixes it.

References

Paper: Wang et al., "RAGEN-2: Reasoning Collapse in Agentic RL", arXiv:2604.06268, 2026
Team: Zihan Wang, Chi Gui, Xing Jin, et al. (advised by Fei-Fei Li, Yejin Choi, Manling Li, et al.)
Institutions: Northwestern, UIUC, Imperial, Oxford, UW, Microsoft, Stanford
Project page: https://ragen-ai.github.io/v2/
Related: RAGEN (predecessor on multi-turn agent self-evolution)