Paper: *When RL Suppresses Its Own Vocabulary: Recovering Reasoning Diversity in Puzzle-to-Math Transfer* Authors: Mayug Maniparambil, Arjun Karuvally, Terrence Sejnowski (Salk Institute/UCSD), Fergal Reid arXiv ID: 2605.29190 (submitted May 28, 2026; cs.LG + cs.CL)
Core finding: RL post-training on puzzles only (no math data) pushes math reasoning to more than double the SFT baseline (16% → 36% on OlymMATH-Hard) — but RL unexpectedly suppresses exploratory primitives like *hypothesize* and *backtrack*. A perplexity-based "novelty bonus" recovers these abilities and adds +7%.
Key points
1. Teaching sudoku, not math
The setup: take a 7B-parameter model, and improve its reasoning without giving it a single math problem — only puzzles: sudoku, logic grids, constraint-satisfaction games, pure "find a feasible solution in a constrained space" training.
Then test it on olympiad math. The result:
- Puzzle SFT: +7 points on OlymMATH-Hard
- RL (GSPO, verifiable rewards): +6 more points
- Novelty bonus: +7 more points
- Total: 16% → 36%, with zero math training data
- define — "let x be the unknown"
- calculate — "2 + 3 = 5"
- constraint — "each row can only appear once"
- hypothesize — "if this cell is 7..."
- verify — "check whether row three satisfies the condition"
- backtrack — "no, try another"
- summarize — "in summary..."
- ...and others
- The paper does not detail the exact puzzle types, counts, or labeling quality used for SFT, affecting reproducibility.
- The 9-primitive taxonomy is self-defined; untracked primitives could be silently enhanced or suppressed by RL.
- Only one 7B model was tested; whether larger models (70B, 405B) exhibit the same vocabulary suppression is unknown.
- 36% on OlymMATH-Hard is still far below top competition-level humans. The claim is not "the model became a math genius" but "non-math data substantially raised the ceiling" — by 20 points.
This is not ordinary transfer learning — it is a cross-domain migration of reasoning ability, like practicing chess for a month and finding your calculus score jumped 20 points.
2. The "primitive vocabulary" of reasoning
The paper's novel methodology: label every span of chain-of-thought with one of nine reasoning *primitives* via a span classifier:
The taxonomy is induced naturally from puzzle solving, which requires repeated *hypothesize → verify → backtrack* loops. This lets the paper answer: what does RL training actually change about how the model reasons?
The answer is unsettling.
3. RL teaches the model to "calculate more" — but "think less"
After puzzle SFT, the model exhibits the full primitive repertoire, including frequent *hypothesize* and *backtrack*.
Then GSPO-based RL continues. Overall performance rises: longer chains, more calculation and verification, more stable solving. But the model also converges: *hypothesize* frequency drops, *backtrack* nearly vanishes. It found a "safe path" — calculate first, verify, never assume, never retreat — and stabilized on it.
Why? RL rewards *correct answers*. Exploration — backtracking after a wrong hypothesis — is *wasted time* in the reward function. Even though such behavior is essential to finding answers, RL only sees the final result.
So RL killed a skill it didn't know it needed. As the paper puts it: "RL suppresses its own vocabulary."
4. The novelty bonus: paying for "different"
The fix: add a term to the reward that penalizes reasoning paths too similar to a reference model's existing solutions — the reference being the post-SFT model that still retains rich exploratory primitives. Similarity is measured via perplexity: low perplexity under the reference model means the rollout follows a well-trodden path (no bonus); high perplexity means a path the reference wouldn't take (bonus).
This redefines the objective from "find one correct answer" to "find multiple *different* correct answers." Effect: +7 points, and *hypothesize* and *backtrack* return — not forced, but guided back by the reward function.
5. Implicit puzzle-to-math transfer
Why does puzzle training help math? Sudoku has no integrals; logic grids have no inequalities.
The data suggests puzzles install domain-agnostic operational skill in general reasoning primitives: *constraint recognition* maps to identifying boundary conditions; *hypothesize-verify-backtrack* maps to "if x = 3 then... no, try x = 5." The bottleneck for a 7B model on hard math isn't knowing the formula — it's systematically exploring the solution space. Puzzles teach the strategy more purely than math data, because math problems carry memorization shortcuts. Puzzles have no formulas to memorize — only pure reasoning.
6. Caveats
7. The blind spot of RL
RL makes the model better at reasoning while destroying some of the most important tools in its reasoning toolbox. This isn't a bug in RL's design — it's RL's philosophy: the reward answers only for outcomes, not for process.
"Getting the right answer" seems like a reliable principle, especially with verifiable rewards. This paper says no: it drives the model onto the shortest path — and the shortest path is not necessarily the one that goes farthest. The novelty bonus is more than a tweak; it redefines the RL objective from "maximize correctness" to "maximize reasoning-path diversity subject to correctness." A patch on RL's philosophy itself.
References:
1. Maniparambil et al., "When RL Suppresses Its Own Vocabulary: Recovering Reasoning Diversity in Puzzle-to-Math Transfer", arXiv:2605.29190, 2026. 2. Shao et al., "DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models", arXiv:2402.03300, 2024. 3. Guo et al., "DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning", arXiv:2501.12948, 2025. 4. Zelikman et al., "STaR: Bootstrapping Reasoning With Reasoning", NeurIPS 2022. 5. Snell et al., "Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters", arXiv:2408.03314, 2024.