Paper Overview
- Field: Machine Learning
- Authors: Ziqi Pang, Yu-Xiong Wang
- Published: 2026-11-02
- arXiv: 2611.03627
- Code: https://github.com/ziqipang/rl_meta_thinking
- Paper: https://arxiv.org/abs/2611.03627
Summary
Reinforcement learning with verifiable rewards (RLVR) can enable generalizable improvement in large language model (LLM) reasoning. This paper studies how to incentivize such generalization through reasoning strategies.
The authors use a controllable equation-solving framework: finding multiple inputs x for equations f(x)=y, where f is a non-bijective string function. This setting makes reasoning strategies observable while supporting a clear definition of easy-to-hard out-of-distribution (OOD) shifts, such as more solutions or longer y.
Key Findings
1. A subtle RLVR failure mode: RLVR teaches the model to solve in-distribution tasks with high accuracy using structured step-by-step derivations. Yet under OOD shifts, the same model abandons derivation and collapses to ineffective brute-force enumeration, causing accuracy to drop sharply.
2. Not a capability or hacking problem: This failure reflects neither insufficient base model capability nor reward hacking—the model succeeds with the correct policy in-distribution. It exposes a deeper challenge in RLVR generalization.
3. An unexpected mitigation: Joint training with bijective inverse tasks significantly recovers the correct policy and accuracy under OOD shifts, even though these bijective tasks are not universally harder and are not closer to the non-bijective OOD evaluations. Their effect is to make derivations more deterministic.
4. Theoretical insight: This effect motivates a theoretical analysis showing that outcome-supervised RLVR alone can shape the reasoning process by implicitly assigning more credit to effective reasoning strategies—even without explicit process supervision.