Does RL Truly Surpass Base Model Reasoning? A Deep Dive into Tsinghua LeapLab's Research

Introduction: Is RLVR the "Master Key" to LLM Reasoning?

Reasoning-oriented LLMs such as OpenAI's o1, DeepSeek-R1, and Kimi-1.5 have made breakthroughs on math and coding tasks. Unlike traditional instruction tuning, Reinforcement Learning with Verifiable Rewards (RLVR) — which replaces costly human feedback with automated rewards (correct math answers, code passing unit tests) — is widely credited as the key driver. Many assumed RLVR lets models self-explore and acquire entirely new reasoning skills like enumeration, self-reflection, and iterative refinement.

A recent study from Tsinghua University's LeapLab challenges this assumption with a fundamental question: Can reinforcement learning really teach LLMs reasoning patterns beyond their base model's capability boundary?

Core Findings: RLVR Improves Efficiency but May Narrow the Reasoning Ceiling

The study uses pass@k with large k to probe the reasoning boundary of base vs. RL-trained models across model families, RL algorithms, and math/code benchmarks. A problem counts as solved if at least one of k samples is correct.

Key results:

• At small k (e.g., k=1), RL-trained models typically outperform base models.
• At large k, base models match or exceed their RL counterparts' pass@k scores.
• Most reasoning paths produced by RL models already exist in the base model's sampling distribution.
• RLVR mainly improves sampling efficiency by biasing outputs toward rewardable paths — at the cost of exploration, narrowing the reasoning boundary relative to the base model.
• Similar trends hold for RLVR-trained visual reasoning tasks.
• Unlike RLVR, knowledge distillation genuinely introduces new knowledge and extends reasoning capability.

Methodology: Quantifying the Reasoning Boundary

The authors extend pass@k from code generation to all verifiable-reward tasks: sample k outputs per problem; the problem scores 1 if at least one passes verification. They use the unbiased estimator of Chen et al. (2021) with n ≥ k samples for low-variance estimation. To guard against lucky guessing in math, they filter guessable problems and manually inspect chain-of-thought outputs. Base models are evaluated with the same prompts as RL models, without few-shot examples.

Experimental Results

Math Reasoning: Base Models Overtake RL Models at Large k

Using Qwen-2.5 (7B/14B/32B) and LLaMA-3.1-8B with GRPO-based zero-RL training, evaluated on GSM8K, MATH500, Minerva, Olympiad, AIME24, and AMC23:

• RL models win at k=1 (higher single-sample accuracy).
• As k grows to dozens or hundreds, base models' pass@k keeps climbing and eventually overtakes RL models. On Minerva, a 32B base model outperforms its RL version by ~9% at k=128.
• Manual CoT analysis on AIME24 shows base models produce longer, more reflective correct reasoning after multiple samples; on GSM8K's hardest problems, success stems from valid reasoning paths, not guessing.

Code Generation: Highly Consistent Results

With CodeR1-Zero-Qwen2.5-7B on LiveCodeBench v5, HumanEval+, and MBPP+, results mirror math: RLVR boosts pass@1, but base models improve faster with larger k and eventually surpass RLVR models, since unit tests make lucky guesses nearly impossible.

Visual Reasoning: Multimodal Tasks Show the Same Pattern

Training Qwen-2.5-VL-7B with EasyR1 and evaluating on filtered MathVista-TestMini and MathVision-TestMini, RLVR again improves small-k performance while base models show broader problem coverage at large k.

Why Doesn't RLVR Create New Reasoning Abilities?

Perplexity analysis shows that high-reward responses from RL models also have high probability density under the base model — the reasoning patterns are not foreign to it. The authors contrast RLVR with classic RL (AlphaGo Zero, DQN), noting two key differences:

1. Massive action space: LLMs' action spaces vastly exceed Go or Atari, making exploration hard. 2. Pretrained priors: RLVR starts from a pretrained model. Training maximizes log-likelihood of reward-bearing responses already in the prior and suppresses others; deviations from the prior mostly yield invalid outputs and negative rewards. Hence the policy stays within the base model's boundary.

Distillation vs. RLVR: What Actually Expands Reasoning

Knowledge distillation from a stronger teacher shifts the student's entire pass@k curve above the base model at all k — a clear contrast with RLVR's boundary-narrowing effect.

Conclusion

RLVR does not teach LLMs reasoning modes beyond their base capabilities; it mainly improves the efficiency of sampling correct paths already encoded in the base model. The reasoning boundary remains capped by the base model. Current RL algorithms also fall short of optimal sampling efficiency. The findings call for new training paradigms — better exploration strategies, or combinations of RLVR with knowledge distillation and supervised learning — to genuinely break through base model reasoning limits.