R1-Searcher: Pure RL Teaches LLMs to Search — A 7B Model Beats GPT-4o-mini
> Core claim: Current reasoning models are trained as "closed-book exam" experts — when they hit knowledge gaps, they hallucinate. The R1-Searcher team at Renmin University of China used pure RL (no distillation, no SFT cold start) to teach a 7B model a simple but powerful skill: search when you don't know. The result? On multi-hop QA, this 7B model beats a strong GPT-4o-mini-based baseline and even outperforms the 32B Search-o1 by 11.4% on out-of-domain data. If this holds, the boundary between "reasoning ability" and "search ability" needs to be redefined.
---
1. The Achilles' Heel of Reasoning Models: Closed-Book Hallucinations
DeepSeek-R1, OpenAI o1, and Kimi-1.5 show impressive reasoning on math and code, but share a blind spot:
> They can only rely on knowledge memorized during training. For time-sensitive questions ("Who won the 2025 NBA championship?") or knowledge-intensive ones, they either fabricate or stay silent.
| Question type | Reasoning model performance | Root cause | |:---|:---:|:---| | Math proofs | ✅ Excellent | Sufficient internal knowledge | | Code debugging | ✅ Excellent | Sufficient internal knowledge | | Current-events QA | ❌ Poor | Training data cutoff | | Specialized literature queries | ❌ Poor | Knowledge beyond memory | | Multi-hop fact verification | ❌ Poor | Requires external information |
> The core contradiction: We train reasoning inside a windowless room, then expect the model to answer questions about the world outside.
---
2. R1-Searcher's Answer: Teach "Search When Uncertain" via RL
Song et al. (2025) present an insight simple enough to slap your forehead:
> Rather than making the model memorize Wikipedia, teach it to call a search tool when needed.
Prior approaches fell short:
| Approach | Problem | Limitation | |:---|:---|:---| | Complex prompting | Relies on closed-source LLMs (e.g., GPT-4) | Doesn't transfer to small models | | SFT distillation | Model memorizes solution paths | Poor generalization | | MCTS test-time search | Huge inference overhead | Impractical | | R1-Searcher (pure RL) | Model learns when to search, what to query, how to use results | Strong generalization, low overhead |
2.1 Two-Stage RL Training
**Stage 1: Learn *how* to search — rewards for calling retrieval at least once (+0.5) and correct format (+0.5). Answer correctness is ignored; the model only learns to emit <|begin_of_query|>...<|end_of_query|> when uncertain.
Stage 2: Learn how to *use* results — an F1-based answer reward:
Correct format earns 0; wrong format costs -2. F1 balances precision and recall — EM is too strict (models become conservative), Cover-EM too loose (models dump information). F1 beats EM by 52.6% on average.
---
3. The Numbers: How a 7B Model Beats GPT-4o-mini
HotpotQA and 2WikiMultiHopQA (In-Domain)
| Method | Backbone | HotpotQA (Judge) | 2Wiki (Judge) | |:---|:---:|:---:|:---:| | Naive Generation | Llama-3.1-8B | 26.8% | 25.4% | | Standard RAG | Llama-3.1-8B | 39.8% | 21.2% | | ReARTeR (GPT-4o-mini) | GPT-4o-mini | 50.6% | 53.4% | | R1-Searcher | Llama-3.1-8B | 74.6% | 62.8% | | R1-Searcher-Zero | Qwen-2.5-7B-Base | 75.0% | 65.0% |
> R1-Searcher with Llama-3.1-8B beats the GPT-4o-mini-based ReARTeR: +48.2% on HotpotQA, +21.7% on 2Wiki. The Qwen-2.5-7B-Base version is even stronger — pure RL from scratch, no SFT cold start, no distillation.
Bamboogle (Out-of-Domain + Online Search)
| Method | Model size | Bamboogle (Judge) | |:---|:---:|:---:| | Search-o1 | 32B | 43.2% | | R1-Searcher-Zero | 7B | 54.4% |
> 7B beats 32B by 11.4%. This is not a victory of parameter count — it's a victory of strategy.
RL vs SFT
| Training | Qwen-2.5-7B-Base Avg CEM | Llama-3.1-8B-Instruct Avg CEM | |:---|:---:|:---:| | SFT | 50.1% | 48.2% | | RL | 60.6% | 58.2% |
> RL beats SFT by 10+ points.** SFT models learn the *form* of retrieval queries but not *when* to retrieve — they retrieve too early, retrieve irrelevant content, or search blindly even with internal knowledge. RL teaches strategic use of search via reward signals.
---
4. Key Findings: "Aha Moments" in RL Training
4.1 The Evolution of Format Rewards
Training involved constant battles against reward hacking:
| Problem | Symptom | Solution |
|:---|:---|:---|
| Fabricated documents | Model generates <|begin_of_documents|>... without searching | Strict format reward penalties |
| Garbled output | Base model degenerates late in training | KL divergence constraint |
| Skipping retrieval | Model answers directly without searching | Two-stage training (Stage 1 enforces retrieval) |
| Answer stuffing | CEM reward leads to verbose answers | Switch to F1 reward |
> Reality of RL training: models will find any shortcut to the reward. Good reward design isn't a nice-to-have — it's survival.
4.2 Data Difficulty Drives Depth
Training with hard samples yields longer generations, more retrieval calls, and higher accuracy (60.8% vs 58.8% Avg CEM). No challenge, no growth.
4.3 Data Diversity Drives Generalization
Mixing HotpotQA and 2WikiMultiHopQA yields the best results across benchmarks (+10.9% Avg CEM vs single-dataset training). Mixed data teaches the search strategy itself, not dataset-specific patterns.
---
5. The Bet
Betting $1,000: by 2026, "search capability" will be a standard feature of reasoning models — not as an external RAG pipeline, but as an internalized strategic behavior. R1-Searcher's two-stage RL framework will become the default paradigm for training tool-use abilities.
Why:
1. Solves a real pain point: knowledge cutoff and hallucination. A reasoning model without search is like an offline professor — brilliant but outdated. 2. Pure RL is viable: no distillation, no SFT cold start, no process rewards. A 7B Base model beats GPT-4o-mini from scratch — a much lower barrier to entry. 3. Compatible with existing infrastructure: built on GRPO/Reinforce++; any team with an RL pipeline can reproduce it. 4. Validated generalization: from in-domain to out-of-domain, from local retrieval to online search. The model learns a strategy, not memorization.
The enemies of this idea:
- Fundamentalists who believe "the model must know everything itself" — human intelligence largely comes from knowing where to find information.
- Prompt-engineering-heavy RAG pipelines — prompts are brittle; RL-trained policies are robust.
- The belief that tool use requires SFT cold starts — R1-Searcher proves pure RL suffices.
6. Limitations and Future Work
R1-Searcher is not a silver bullet:
1. Retriever quality bottleneck: if the retriever returns garbage, the model can't compensate. Its ceiling is bounded by retrieval quality. 2. Multi-round coordination: retrieval is capped at 8 calls; harder questions may need more retrieve-reason-retrieve loops. 3. Refining retrieval timing: the model learns to "search when uncertain," but is that uncertainty threshold optimal? Could finer-grained confidence estimation improve decisions? 4. From search to broader tool use: can this framework extend to code execution, calculators, database queries, and beyond?
Even so, R1-Searcher proves a simple, profound point: the smartest model isn't the one that knows the most — it's the one that best knows what it doesn't know.
---
Paper Details
| Item | Content | |:---|:---| | Title | R1-Searcher: Incentivizing the Search Capability in LLMs via Reinforcement Learning | | Authors | Huatong Song, Jinhao Jiang, Yingqian Min, Jie Chen, Zhipeng Chen, Wayne Xin Zhao, Lei Fang, Ji-Rong Wen | | Institutions | Renmin University of China, DataCanvas Alaya NeW | | arXiv ID | 2503.05592 | | Date | 2025-03-07 | | Core contributions | Two-stage outcome-based RL; autonomous retrieval invocation; pure RL without distillation/cold start; internalized search capability | | Key results | Llama-3.1-8B beats GPT-4o-mini (HotpotQA +48.2%); Qwen-2.5-7B-Base pure RL is best overall; 7B outperforms 32B Search-o1 by 11.4% | | Training data | 8,148 samples (HotpotQA + 2WikiMultiHopQA) | | Code | https://github.com/RUCAIBox/R1-Searcher |