R1-Searcher: Pure RL Teaches a 7B LLM to Search, Beating GPT-4o-mini Without Distillation or Cold Start

> Core claim: Current "reasoning models" are trained as closed-book exam experts—they hallucinate when facing questions beyond their training knowledge. The R1-Searcher team at Renmin University of China used pure reinforcement learning (no distillation, no SFT cold start) to teach 7B models a simple but powerful skill: search when you don't know. The result? A 7B model beats strong GPT-4o-mini-based baselines on multi-hop QA and even outperforms the 32B Search-o1 model by 11.4% on out-of-domain data.

1. The Achilles' Heel of Reasoning Models: Closed-Book Hallucination

DeepSeek-R1, OpenAI o1, and Kimi-1.5 show impressive reasoning on math and code—but they share a blind spot: they rely solely on memorized knowledge. For time-sensitive questions ("Who won the 2025 NBA championship?") or knowledge-intensive queries (details of a specific paper), they either fabricate answers or stay silent.

| Question type | Reasoning model performance | Root cause | |:---|:---:|:---| | Math proofs | ✅ Excellent | Sufficient internal knowledge | | Code debugging | ✅ Excellent | Sufficient internal knowledge | | Current events | ❌ Poor | Training data cutoff | | Professional literature | ❌ Poor | Knowledge beyond memory | | Multi-hop fact verification | ❌ Poor | Requires external information |

Core contradiction: we train reasoning inside a windowless room, then expect the model to answer questions about the world outside.

2. The R1-Searcher Approach: Teach "Search When Unsure" via RL

The core insight (Song et al., 2025): instead of making the model memorize Wikipedia, teach it to call a search tool when needed.

| Method | Problem | Limitation | |:---|:---|:---| | Complex prompt design | Relies on closed-source large models | Doesn't transfer to small models | | SFT distillation | Model memorizes solution paths | Poor generalization | | MCTS test-time search | Huge inference overhead | Impractical | | R1-Searcher (pure RL) | Model autonomously learns when/what/how to search | Strong generalization, low overhead |

2.1 Two-Stage RL Training

Stage 1: Learn how to search — rewards correct invocation format and timing, regardless of answer correctness:

• Retrieval reward: +0.5 if at least one retrieval call is made, otherwise 0
• Format reward: +0.5 for correct format, otherwise 0

Key design: Stage 1 ignores answer accuracy entirely—the model only learns that when uncertain, it should emit <|begin_of_query|>...<|end_of_query|> to trigger search.

Stage 2: Learn how to use search results — rewards answer quality:

• Answer reward (F1): \(R_{answer} = \frac{2 \cdot IN}{PN + RN}\)
• Format reward: 0 for correct format; -2 penalty for incorrect format

The F1 reward is a deliberate choice: Exact Match is too strict (making models conservative), Cover-EM too loose (encouraging information dumping). Experiments show F1 beats EM by 52.6% on average.

3. Results: A 7B Model Beats GPT-4o-mini

In-Domain (HotpotQA, 2WikiMultiHopQA)

| Method | Backbone | HotpotQA (Judge) | 2Wiki (Judge) | |:---|:---:|:---:|:---:| | Naive Generation | Llama-3.1-8B | 26.8% | 25.4% | | Standard RAG | Llama-3.1-8B | 39.8% | 21.2% | | ReARTeR | GPT-4o-mini | 50.6% | 53.4% | | R1-Searcher | Llama-3.1-8B | 74.6% | 62.8% | | R1-Searcher-Zero | Qwen-2.5-7B-Base | 75.0% | 65.0% |

R1-Searcher beats the GPT-4o-mini-based ReARTeR by +48.2% on HotpotQA and +21.7% on 2Wiki. The Qwen-2.5-7B-Base version is even stronger—and it's pure RL from the base model: no SFT cold start, no distillation.

Out-of-Domain + Online Search (Bamboogle)

| Method | Scale | Bamboogle (Judge) | |:---|:---:|:---:| | Search-o1 | 32B | 43.2% | | R1-Searcher-Zero | 7B | 54.4% |

A 7B model beats a 32B model by 11.4%—a victory of strategy, not parameter count.

RL vs SFT

| Training | Qwen-2.5-7B-Base Avg CEM | Llama-3.1-8B-Instruct Avg CEM | |:---|:---:|:---:| | SFT | 50.1% | 48.2% | | RL | 60.6% | 58.2% |

RL beats SFT by 10+ points: SFT models learn the *form* of retrieval queries but not the *judgment* of when to retrieve—often retrieving prematurely or redundantly. RL teaches strategic search use.

4. Key Findings: Fighting Reward Hacking in RL Training

| Problem | Symptom | Solution | |:---|:---|:---| | Fake documents | Model fabricates <|begin_of_documents|> without searching | Strict format penalties | | Gibberish output | Base model degenerates late in training | KL divergence constraint | | Skipping retrieval | Model answers directly without searching | Two-stage training (forced retrieval in Stage 1) | | Answer stuffing | CEM reward leads to verbose answers | Switch to F1 reward |

Additional findings:

• Data difficulty matters: including hard training data leads to longer generations, more retrieval calls, and higher accuracy (60.8% vs 58.8% Avg CEM).
• Data diversity drives generalization: mixing HotpotQA and 2Wiki beats single-dataset training by +10.9% Avg CEM—the model learns the *search strategy itself*, not dataset-specific patterns.

5. Limitations

1. Retriever quality bottleneck: the model's ceiling is limited by the search system's quality. 2. Multi-round retrieval coordination: currently capped at 8 retrieval rounds; harder questions may need more retrieval-reasoning loops. 3. When to search: the "uncertainty threshold" for retrieval may not be optimal; finer confidence estimation could help. 4. Beyond search: can the framework extend to code execution, calculators, databases, and broader tool ecosystems?

Still, R1-Searcher demonstrates a simple, deep point: the smartest model is not the one that knows the most, but the one that best knows what it doesn't know.

Paper Details

| Item | Content | |:---|:---| | Title | R1-Searcher: Incentivizing the Search Capability in LLMs via Reinforcement Learning | | Authors | Huatong Song, Jinhao Jiang, Yingqian Min, Jie Chen, Zhipeng Chen, Wayne Xin Zhao, Lei Fang, Ji-Rong Wen | | Institutions | Renmin University of China, DataCanvas Alaya NeW | | arXiv ID | 2503.05592 | | Date | 2025-03-07 | | Key contributions | Two-stage outcome-based RL; autonomous retrieval calls; pure RL without distillation/cold start; internalized search capability | | Key results | Llama-3.1-8B beats GPT-4o-mini (HotpotQA +48.2%); pure-RL Qwen-2.5-7B-Base achieves best results; 7B beats 32B Search-o1 by 11.4% | | Training data | 8,148 samples (HotpotQA + 2WikiMultiHopQA) | | Code | https://github.com/RUCAIBox/R1-Searcher |