When RL Kills Its Own Vocabulary: How Reinforcement Learning Suppresses Model Exploration

Forum topic · 小凯 · 2026-05-31

Summary

A forum post discusses the paper 'When RL Suppresses Its Own Vocabulary: Recovering Reasoning Diversity in Puzzle-to-Math Transfer' (arXiv:2605.29190). The study trains a 7B-parameter language model exclusively on constraint-satisfaction puzzles—no math data—and achieves remarkable transfer: SFT on puzzles lifts OlymMATH-Hard accuracy from 16% to 23%, RL with verifiable rewards adds another 6 points to 29%, and a novelty bonus pushes it to 36%, more than doubling the original baseline. The authors introduce a 9-class taxonomy of reasoning primitives (define, calculate, hypothesize, verify, backtrack, etc.) and track how they evolve during training. Key finding: RL improves overall performance but inadvertently suppresses exploratory primitives—hypothesizing and backtracking nearly disappear as the model converges on a safe 'calculate-then-verify' strategy, since outcome-only rewards treat exploration as wasted effort. The proposed fix is a perplexity-based novelty bonus that rewards rollouts dissimilar to the reference model's typical solutions, restoring diversity and adding +7%. The post also notes limitations: unspecified puzzle dataset details, possible incompleteness of the primitive taxonomy, single 7B model tested, and modest absolute accuracy.

Paper: When RL Suppresses Its Own Vocabulary: Recovering Reasoning Diversity in Puzzle-to-Math Transfer Authors: Mayug Maniparambil, Arjun Karuvally, Terrence Sejnowski, Fergal Reid arXiv: 2605.29190 (cs.LG, cs.CL)

Key Points

Puzzle-only training transfers to math: A 7B model post-trained solely on constraint-satisfaction puzzles (SFT) improves OlymMATH-Hard accuracy from 16% to 23%. Adding RL with verifiable rewards (GSPO) raises it to 29%, and a novelty bonus pushes it to 36%—all without ever showing the model a math problem.
Reasoning primitives: The authors use a 9-class span classifier to label chain-of-thought segments into primitives: define, calculate, constraint identification, hypothesize, verify, backtrack, summarize, and more. Puzzle solving naturally exercises the hypothesize → verify → backtrack loop.
RL teaches calculating more, but thinking less: After RL, overall accuracy improves and chains get longer with more calculation/verification steps. But the *hypothesize* primitive declines and *backtrack* almost vanishes. Outcome-only rewards make exploration look like wasted time—RL optimizes away a skill it needs, or as the paper puts it, "RL suppresses its own vocabulary."
Novelty bonus as a fix: The reward function adds a term penalizing rollouts too similar (measured by perplexity under the puzzle-SFT reference model) to existing solutions. High perplexity—unfamiliar reasoning paths—earns a bonus. This restores hypothesizing/backtracking and yields an additional +7 points.
Why puzzles work: Primitives like constraint identification and hypothesis-backtracking are domain-agnostic. Puzzles teach pure reasoning strategy without the formula-memorization shortcuts that contaminate math data.

Limitations Noted in the Post

The exact puzzle dataset composition is not fully specified, affecting reproducibility.
The 9-primitive taxonomy is author-defined and may miss other important reasoning operations.
Experiments cover only a 7B model; whether larger models exhibit the same suppression is untested.
36% on OlymMATH-Hard is still far from elite competition performance—the contribution is the direction: reasoning training may not require in-domain data.

Takeaway

Reinforcement learning made the model better at reasoning while destroying some of its most important reasoning tools. "Getting the right answer" as the sole objective pushes models onto the shortest path, which is not necessarily the path that goes furthest. The novelty bonus redefines the objective: maximize solution diversity while maintaining correctness—a patch on the reward-only-outcomes philosophy of RL.

References:

1. Maniparambil et al., "When RL Suppresses Its Own Vocabulary: Recovering Reasoning Diversity in Puzzle-to-Math Transfer", arXiv:2605.29190, 2026. 2. Shao et al., "DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models", arXiv:2402.03300, 2024. 3. Guo et al., "DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning", arXiv:2501.12948, 2025. 4. Zelikman et al., "STaR: Bootstrapping Reasoning With Reasoning", NeurIPS 2022. 5. Snell et al., "Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters", arXiv:2408.03314, 2024.