Does RLVR Really Exceed Base Model Reasoning Limits? Tsinghua LeapLab Study Explained

Forum topic · QianXun · 2025-11-24

Summary

A study from Tsinghua University's LeapLab challenges the common belief that Reinforcement Learning with Verifiable Rewards (RLVR) enables large language models to acquire reasoning abilities beyond their base models. Using pass@k evaluation with large k values across math, code generation, and visual reasoning benchmarks, the researchers found that RL-trained models outperform base models only at small k. As sampling attempts increase, base models match or surpass RL-trained models—for example, a 32B base model scored roughly 9% higher than its RL counterpart at k=128 on Minerva. Perplexity analysis shows that reasoning paths generated by RL models largely already exist in the base model's output distribution: RLVR mainly reweights the distribution toward rewardable responses, improving sampling efficiency but narrowing the exploration boundary. The paper contrasts this with knowledge distillation, which can genuinely introduce new reasoning patterns and extend the capability boundary beyond the base model. The findings suggest RLVR's value should be reconsidered and motivate new training paradigms for advancing LLM reasoning.

Introduction

Reasoning-focused LLMs such as OpenAI's o1, DeepSeek-R1, and Kimi-1.5 are widely credited to Reinforcement Learning with Verifiable Rewards (RLVR), which replaces costly human feedback with automated rewards (e.g., verified math answers or passing unit tests). Many assume RLVR lets models self-explore entirely new reasoning skills. A recent study from Tsinghua University's LeapLab questions this assumption, asking: can RL actually teach LLMs reasoning patterns beyond their base model's capability boundary?

Core Findings

The study uses pass@k with large k values to probe reasoning boundaries across model families, RL algorithms, and math/code benchmarks.
RLVR does not produce fundamentally new reasoning patterns. RL-trained models win at small k (e.g., k=1), but at large k, base models reach or exceed their RL counterparts' pass@k scores.
On Minerva, a 32B base model outperformed its RL-trained version by about 9% at k=128.
Most reasoning paths of RL models already exist in the base model's sampling distribution. RL mainly shifts the output distribution toward rewardable paths, improving sampling efficiency while sacrificing exploration and narrowing the reasoning boundary.
Similar trends hold for visual reasoning tasks trained with RLVR.
Knowledge distillation, unlike RLVR, genuinely introduces new knowledge and expands the reasoning boundary beyond the base model.

Methodology

pass@k is extended from code generation to all verifiable-reward tasks: a problem counts as solved if at least one of k samples passes verification.
Low-variance estimation follows Chen et al. (2021): generate n ≥ k samples per problem and estimate pass@k from correct counts.
To guard against lucky guesses in math, easily guessable problems were filtered out and chain-of-thought outputs manually inspected.
Experiments cover Qwen-2.5 (7B/14B/32B), LLaMA-3.1-8B, GRPO-based zero-RL training, benchmarks including GSM8K, MATH500, Minerva, Olympiad, AIME24, AMC23, LiveCodeBench v5, HumanEval+, MBPP+, and filtered MathVista/MathVision for visual reasoning. Base models used identical prompts without few-shot examples for fairness.

Experimental Results

Math Reasoning

Consistent pattern across benchmarks: RL models lead at small k; base models overtake RL models as k grows to tens or hundreds.
Manual CoT analysis on AIME24 shows base models produce longer, more reflective correct reasoning after multiple samples.
Analysis of hardest GSM8K problems confirms success comes from valid reasoning paths, not guessing.

Code Generation

Using Code-R1's CodeR1-Zero-Qwen2.5-7B on LiveCodeBench v5, HumanEval+, and MBPP+, results mirror math: RLVR improves pass@1, but base models surpass RL models at large k, since guessing is nearly impossible with full unit-test verification.

Visual Reasoning

Training Qwen-2.5-VL-7B with EasyR1 and evaluating on filtered MathVista-TestMini and MathVision-TestMini shows the same trend: RLVR helps at small k, but base models cover a broader problem space at large k.

Why RLVR Fails to Add New Reasoning

Perplexity analysis shows high-reward responses from RL models also have high probability density under the base model's distribution. Unlike classic RL (AlphaGo Zero, DQN), RLVR for LLMs faces:

1. Huge action spaces, making exploration of reward signals difficult. 2. Pretrained priors: RLVR maximizes the likelihood of rewardable responses already in the prior and suppresses the rest. Deviations from the prior likely yield invalid outputs and negative rewards, so the policy stays within the base model's boundary.

Distillation as the True Boundary-Expander

Knowledge distillation from a stronger teacher yields student pass@k curves above the base model at all k values—directly contrasting RLVR's limitation and highlighting distillation's unique role in injecting new reasoning patterns.

Conclusion

RLVR primarily improves the efficiency of sampling correct paths already encoded in the base model rather than teaching new reasoning modes; the capability boundary remains set by the base model. The authors call for better exploration strategies and alternative paradigms (e.g., combining distillation or supervised learning) to genuinely push past base-model reasoning limits.