RAGEN-2: Template Collapse in Agentic RL — When LLM Agents Master 'Correct Nonsense'

Forum topic · 小凯 · 2026-06-20

Summary

A Chinese forum post analyzes RAGEN-2, a research paper from teams advised by Fei-Fei Li and Yejin Choi, describing a new failure mode in multi-turn LLM agent reinforcement learning called Template Collapse. The model produces outputs that look diverse and well-structured but are essentially independent of the input prompt — 'correct nonsense.' Standard monitoring metrics fail to catch this: entropy stays high, rewards look moderate, and format checks pass. The paper proposes mutual information (MI) between input and reasoning chain as the proper diagnostic, decomposing reasoning quality into conditional entropy H(Z|X) and MI I(X;Z). Empirically, MI correlates +0.39 (Spearman) with final task performance while entropy correlates negatively (-0.11 to -0.14). The authors explain the mechanism via a signal-to-noise ratio (SNR) gradient decomposition: as prompt-level reward variance collapses, input-agnostic regularization (KL, entropy) dominates updates, erasing cross-input differences. Their fix, SNR-Aware Filtering, keeps only the top-ρ prompts by reward variance each iteration and is a lightweight, pluggable addition to PPO/GRPO-style training. Validated on Sokoban, math, web navigation, code, and tool-use tasks, it consistently improves performance and input dependence. Limitations include single-agent scope and possible reward-variance gaming. Project page: https://ragen-ai.github.io/v2/ (arXiv:2604.06268).

RAGEN-2: Template Collapse — The Hidden Crisis When AI Learns 'Correct Nonsense'

English translation and structured summary of a Chinese forum post discussing the RAGEN-2 paper (teams advised by Fei-Fei Li, Yejin Choi, and Manling Li; Northwestern, UIUC, Imperial, Oxford, UW, Microsoft, Stanford).

Key points

Template Collapse defined: LLM agents trained with multi-turn RL can silently learn a universal template — answers look rich, fluent, and varied but bear almost no relation to the input prompt. The model stops reasoning and starts reciting.
High entropy ≠ real reasoning: The community's iron rule (high entropy = healthy training) breaks down. A model can keep stable or high entropy while completely ceasing input-dependent thought. Template collapse passes every existing check: entropy looks normal (diversity within one input), sparse rewards look moderate (the template guesses right ~50% of the time), format checks pass, and reasoning-length decline is too slow a signal.
Mutual information catches it: RAGEN-2 decomposes reasoning quality into conditional entropy H(Z|X) (diversity for the same input) and mutual information I(X;Z) (does output actually depend on input?). Template collapse = high H(Z|X) + low I(X;Z). Since exact MI is intractable online, lightweight proxies are proposed: matching each reasoning chain Zᵢ against all inputs in a batch via matched − marginal = log P(Zᵢ|Xᵢ) − log P(Zᵢ|X_k), with variants like Retrieval-Acc and MI-ZScore-EMA.
MI predicts performance; entropy does not: Trajectory-level MI-ZScore correlates +0.39 (Spearman) with final performance, while entropy correlates −0.11 to −0.14 — entropy's trend runs *opposite* to performance. MI also drops earlier than task reward, acting as an early-warning signal.

Why collapse happens: the SNR view

Each RL gradient update decomposes as g_total = g_signal + g_task-noise + g_reg, where only the regularization term (KL + entropy, applied uniformly per chain regardless of prompt) is controllable. When prompt-level reward variance collapses (increasingly many prompts yield near-identical rewards no matter the reasoning), the task signal vanishes while regularization keeps pushing updates in an input-agnostic direction: low SNR → erasure of cross-input differences → template collapse.

The fix: SNR-Aware Filtering

Per iteration: sample G rollouts per prompt, compute reward variance, keep only the top-ρ prompts by variance, and update the policy on those alone. It requires no extra models, no extra inference, and plugs into PPO/GRPO/REINFORCE++. Reward variance correlates +0.63 with task reward but only weakly with entropy (−0.14) and length (+0.12), i.e., it is an independent signal dimension. Filtering amplifies signal where KL/entropy tuning only reduces noise — the two are complementary.

Validated across planning (Sokoban), math (MetaMath), web navigation, code execution, and tool use, filtering consistently improves both task performance and input dependence (MI).

Behavioral signatures and limitations

Reasoning length declines monotonically across 8 environments after RL training — not learned conciseness, but templating in fewer words.
Perfect format compliance (e.g., valid JSON) guarantees nothing about input dependence.
Multi-turn tasks hide the failure: early templated mistakes get corrected later, and final rewards can still be positive.
Paper-stated limitations: the SNR decomposition assumes separable signal/noise; only single-agent settings are studied; models might game the variance filter; variance is unreliable in sparse/noisy-reward settings; aggressive filtering may shrink exploration.

Bottom line

LLM agents in RL training can learn 'correct nonsense' — superficially diverse, structurally complete, but input-independent. Entropy cannot see it; mutual information can. Collapsed reward variance is the root cause, and filtering by variance fixes it.

References (as cited in the post)

Paper: Wang et al., "RAGEN-2: Reasoning Collapse in Agentic RL", arXiv:2604.06268, 2026
Project page: https://ragen-ai.github.io/v2/
Related: RAGEN (predecessor work on multi-turn agent self-evolution)