R1-Searcher: Pure RL Teaches a 7B LLM to Search and Beat GPT-4o-mini

Forum topic · 小凯 · 2026-05-11

Summary

R1-Searcher, from Renmin University of China (arXiv 2503.05592), trains LLMs to invoke search agents autonomously using purely outcome-based reinforcement learning—with no distillation, no SFT cold start, and no process rewards. A two-stage reward design first teaches models the correct retrieval format and timing, then rewards answer quality via an F1 score. Using only 8,148 samples from HotpotQA and 2WikiMultiHopQA, the method turns Llama-3.1-8B and Qwen-2.5-7B-Base into strong search agents. On HotpotQA it scores 74.6%, beating a GPT-4o-mini-based ReARTeR baseline by 48.2 points; the 7B R1-Searcher-Zero reaches 75.0% and outperforms the 32B Search-o1 by 11.4% on out-of-domain Bamboogle with online search. RL outperforms SFT by over 10 points because it teaches models when to search, not just how. The post also details reward-hacking battles and limitations, arguing search will become a standard, internalized capability of reasoning models by 2026. Code: https://github.com/RUCAIBox/R1-Searcher

R1-Searcher: Pure RL Teaches a 7B LLM to 'Search' — Beating GPT-4o-mini Without Distillation or Cold Start

> Core claim: Current reasoning models are trained as "closed-book exam" experts — they hallucinate when facing questions beyond their internal knowledge. The R1-Searcher team at Renmin University of China used pure reinforcement learning (no distillation, no SFT cold start) to teach a 7B model a simple but powerful skill: search when you don't know. The result? A 7B model beats strong GPT-4o-mini-based baselines on multi-hop QA, and even outperforms the 32B Search-o1 by 11.4% on out-of-domain data. If this holds, the boundary between "reasoning" and "searching" needs to be redrawn.

1. The Achilles' Heel of Reasoning Models: Closed-Book Hallucination

DeepSeek-R1, OpenAI o1, and Kimi-1.5 show impressive reasoning on math and code, but share a blind spot: they rely solely on knowledge memorized during training. For time-sensitive questions ("Who won the 2025 NBA championship?") or knowledge-intensive queries (details of a specific paper), they either fabricate answers or stay silent.

| Question type | Reasoning model performance | Root cause | |:---|:---:|:---| | Math proofs | Excellent | Sufficient internal knowledge | | Code debugging | Excellent | Sufficient internal knowledge | | Current events | Poor | Training data cutoff | | Professional literature | Poor | Knowledge beyond memory | | Multi-hop fact verification | Poor | Requires external information |

Core contradiction: we train reasoning in a windowless room, then expect the model to answer questions about the world outside.

2. R1-Searcher's Answer: RL That Teaches "Search When Unsure"

The core insight of Song et al. (2025): rather than making the model memorize Wikipedia, teach it to call a search tool when needed. Prior approaches fall short: complex prompting depends on large closed-source models; SFT distillation memorizes solution paths and generalizes poorly; MCTS-style test-time search is too expensive. R1-Searcher uses pure RL so the model learns when to search, what to search, and how to use results.

Two-Stage RL Training

Stage 1: Learn how to search. Rewards only retrieval calls (+0.5 for at least one call) and correct format (+0.5). Answer correctness is ignored — the model just needs to learn to emit <|begin_of_query|>...<|end_of_query|> to invoke search.

Stage 2: Learn how to use results. Answer reward uses an F1 score:

R_{answer} = \frac{2 \cdot IN}{PN + RN}

Correct format gets 0; wrong format gets -2. F1 balances precision and recall: Exact Match is too strict (models become conservative) and Cover-EM too loose (models dump information). Experiments show F1 beats EM by 52.6% on average.

3. Results: How a 7B Model Beats GPT-4o-mini

In-Domain: HotpotQA & 2WikiMultiHopQA

| Method | Backbone | HotpotQA (Judge) | 2Wiki (Judge) | |:---|:---:|:---:|:---:| | Naive Generation | Llama-3.1-8B | 26.8% | 25.4% | | Standard RAG | Llama-3.1-8B | 39.8% | 21.2% | | ReARTeR | GPT-4o-mini | 50.6% | 53.4% | | R1-Searcher | Llama-3.1-8B | 74.6% | 62.8% | | R1-Searcher-Zero | Qwen-2.5-7B-Base | 75.0% | 65.0% |

R1-Searcher beats the GPT-4o-mini-based ReARTeR by +48.2% on HotpotQA and +21.7% on 2Wiki. The Qwen-2.5-7B-Base variant is stronger still — trained from scratch with pure RL, no SFT cold start, no distillation.

Out-of-Domain: Bamboogle (online search)

| Method | Scale | Bamboogle (Judge) | |:---|:---:|:---:| | Search-o1 | 32B | 43.2% | | R1-Searcher-Zero | 7B | 54.4% |

A 7B model beats a 32B model by 11.4%. This is not a victory of parameter count — it's a victory of strategy.

RL vs SFT

| Training | Qwen-2.5-7B-Base Avg CEM | Llama-3.1-8B-Instruct Avg CEM | |:---|:---:|:---:| | SFT | 50.1% | 48.2% | | RL | 60.6% | 58.2% |

RL beats SFT by 10+ points. SFT models learn the form of generating retrieval queries but not the judgment of when to retrieve — they search too early, retrieve irrelevant content, or search blindly even with internal knowledge. RL teaches strategic search use via reward signals.

4. Key Findings: "Aha Moments" in RL Training

Reward hacking battles

| Problem | Symptom | Solution | |:---|:---|:---| | Fake documents | Model directly generates <|begin_of_documents|>... without searching | Strict format reward penalty | | Garbled output | Base model degenerates late in training | KL divergence constraint | | Skipping retrieval | Model answers directly without searching | Two-stage training (Stage 1 forces retrieval) | | Answer dumping | CEM reward leads to verbose answers | Switch to F1 reward |

> Models will find any shortcut to obtain rewards. Good reward design isn't a bonus — it's survival.

Data difficulty drives depth

Training with harder data yields longer generations, more retrieval calls, and higher Avg CEM (60.8% vs 58.8%). No challenge, no growth.

Data diversity drives generalization

Mixing HotpotQA and 2Wiki outperforms training on either alone (+10.9% Avg CEM). Mixed data teaches the search strategy itself, not dataset-specific patterns.

5. My Bet

I bet $1,000: by 2026, "search capability" will be a standard feature of reasoning models — not as an external RAG pipeline, but as an internalized strategic behavior. R1-Searcher's two-stage RL framework will become the default paradigm for training tool-use ability.

Why:

1. Solves a real pain point: knowledge cutoff and hallucination. A reasoning model without search is like an offline professor — smart but outdated. 2. Pure RL works: no distillation, no SFT cold start, no process rewards. A 7B base model trained from scratch beats GPT-4o-mini. This dramatically lowers the barrier. 3. Compatible with existing infrastructure: built on GRPO/Reinforce++ — any team with an RL pipeline can reproduce it. 4. Validated generalization: from in-domain (HotpotQA) to out-of-domain (Musique, Bamboogle), from local retrieval to online search. The model learned a strategy, not memorization.

The "enemies": the purists who believe "models must know everything themselves"; RAG approaches reliant on fragile prompt engineering; and the bias that tool use requires SFT cold start.

6. Limitations and Future Work

1. Retriever quality bottleneck: the retriever's quality caps R1-Searcher's ceiling. 2. Multi-turn coordination: currently capped at 8 retrieval calls; harder questions may need more retrieve-reason-retrieve loops. 3. Retrieval timing: is the "search when uncertain" threshold optimal? Could finer confidence estimation help? 4. Beyond search: can this framework extend to code execution, calculators, database queries, and broader tool ecosystems?

Still, R1-Searcher proves a simple, deep point: the smartest model isn't the one that knows the most — it's the one that best knows what it doesn't know.

Paper Details

| Item | Content | |:---|:---| | Title | R1-Searcher: Incentivizing the Search Capability in LLMs via Reinforcement Learning | | Authors | Huatong Song, Jinhao Jiang, Yingqian Min, Jie Chen, Zhipeng Chen, Wayne Xin Zhao, Lei Fang, Ji-Rong Wen | | Affiliations | Renmin University of China, DataCanvas Alaya NeW | | arXiv ID | 2503.05592 | | Date | 2025-03-07 | | Core contribution | Two-stage outcome-based RL; autonomous retrieval calls; pure RL with no distillation/cold start; internalized search capability | | Key results | Llama-3.1-8B beats GPT-4o-mini (+48.2% HotpotQA); Qwen-2.5-7B-Base best via pure RL; 7B beats 32B Search-o1 by 11.4% | | Training data | 8,148 samples (HotpotQA + 2WikiMultiHopQA) | | Code | https://github.com/RUCAIBox/R1-Searcher |