English static mirror for SEO/GEO · AI-assisted translation · Read Chinese original

R1-Searcher: Pure RL Teaches a 7B LLM to Search, Beating GPT-4o-mini Without Distillation or Cold Start

Forum topic · 小凯 · 2026-05-11

Summary

R1-Searcher, from Renmin University of China (arXiv:2503.05592), trains LLMs to invoke search engines autonomously using a two-stage outcome-based reinforcement learning framework—no distillation, no supervised fine-tuning cold start. Stage 1 rewards correct retrieval call format and timing; Stage 2 rewards answer quality via an F1-based reward, which outperforms Exact Match by 52.6% on average. Trained on only 8,148 samples from HotpotQA and 2WikiMultiHopQA, the resulting Llama-3.1-8B model beats a GPT-4o-mini-based ReARTeR baseline on multi-hop QA (+48.2% on HotpotQA, +21.7% on 2Wiki), while a pure-RL Qwen-2.5-7B-Base variant (R1-Searcher-Zero) outperforms the 32B Search-o1 model by 11.4% on out-of-domain Bamboogle with online search. RL training also outperforms SFT by over 10 points, as it teaches models when to search rather than just how to imitate search queries. The authors combat reward hacking (fake documents, skipped retrieval, answer stuffing) via format penalties, KL constraints, and staged training. Code is available at https://github.com/RUCAIBox/R1-Searcher.

R1-Searcher: Pure RL Teaches a 7B LLM to Search, Beating GPT-4o-mini Without Distillation or Cold Start

> Core claim: Current "reasoning models" are trained as closed-book exam experts—they hallucinate when facing questions beyond their training knowledge. The R1-Searcher team at Renmin University of China used pure reinforcement learning (no distillation, no SFT cold start) to teach 7B models a simple but powerful skill: search when you don't know. The result? A 7B model beats strong GPT-4o-mini-based baselines on multi-hop QA and even outperforms the 32B Search-o1 model by 11.4% on out-of-domain data.

1. The Achilles' Heel of Reasoning Models: Closed-Book Hallucination

DeepSeek-R1, OpenAI o1, and Kimi-1.5 show impressive reasoning on math and code—but they share a blind spot: they rely solely on memorized knowledge. For time-sensitive questions ("Who won the 2025 NBA championship?") or knowledge-intensive queries (details of a specific paper), they either fabricate answers or stay silent.

| Question type | Reasoning model performance | Root cause | |:---|:---:|:---| | Math proofs | ✅ Excellent | Sufficient internal knowledge | | Code debugging | ✅ Excellent | Sufficient internal knowledge | | Current events | ❌ Poor | Training data cutoff | | Professional literature | ❌ Poor | Knowledge beyond memory | | Multi-hop fact verification | ❌ Poor | Requires external information |

Core contradiction: we train reasoning inside a windowless room, then expect the model to answer questions about the world outside.

2. The R1-Searcher Approach: Teach "Search When Unsure" via RL

The core insight (Song et al., 2025): instead of making the model memorize Wikipedia, teach it to call a search tool when needed.

| Method | Problem | Limitation | |:---|:---|:---| | Complex prompt design | Relies on closed-source large models | Doesn't transfer to small models | | SFT distillation | Model memorizes solution paths | Poor generalization | | MCTS test-time search | Huge inference overhead | Impractical | | R1-Searcher (pure RL) | Model autonomously learns when/what/how to search | Strong generalization, low overhead |

2.1 Two-Stage RL Training

Stage 1: Learn how to search — rewards correct invocation format and timing, regardless of answer correctness:

  • Retrieval reward: +0.5 if at least one retrieval call is made, otherwise 0
  • Format reward: +0.5 for correct format, otherwise 0
  • Key design: Stage 1 ignores answer accuracy entirely—the model only learns that when uncertain, it should emit <|begin_of_query|>...<|end_of_query|> to trigger search.

    Stage 2: Learn how to use search results — rewards answer quality:

  • Answer reward (F1): \(R_{answer} = \frac{2 \cdot IN}{PN + RN}\)
  • Format reward: 0 for correct format; -2 penalty for incorrect format
  • The F1 reward is a deliberate choice: Exact Match is too strict (making models conservative), Cover-EM too loose (encouraging information dumping). Experiments show F1 beats EM by 52.6% on average.

    3. Results: A 7B Model Beats GPT-4o-mini

    In-Domain (HotpotQA, 2WikiMultiHopQA)

    | Method | Backbone | HotpotQA (Judge) | 2Wiki (Judge) | |:---|:---:|:---:|:---:| | Naive Generation | Llama-3.1-8B | 26.8% | 25.4% | | Standard RAG | Llama-3.1-8B | 39.8% | 21.2% | | ReARTeR | GPT-4o-mini | 50.6% | 53.4% | | R1-Searcher | Llama-3.1-8B | 74.6% | 62.8% | | R1-Searcher-Zero | Qwen-2.5-7B-Base | 75.0% | 65.0% |

    R1-Searcher beats the GPT-4o-mini-based ReARTeR by +48.2% on HotpotQA and +21.7% on 2Wiki. The Qwen-2.5-7B-Base version is even stronger—and it's pure RL from the base model: no SFT cold start, no distillation.

    Out-of-Domain + Online Search (Bamboogle)

    | Method | Scale | Bamboogle (Judge) | |:---|:---:|:---:| | Search-o1 | 32B | 43.2% | | R1-Searcher-Zero | 7B | 54.4% |

    A 7B model beats a 32B model by 11.4%—a victory of strategy, not parameter count.

    RL vs SFT

    | Training | Qwen-2.5-7B-Base Avg CEM | Llama-3.1-8B-Instruct Avg CEM | |:---|:---:|:---:| | SFT | 50.1% | 48.2% | | RL | 60.6% | 58.2% |

    RL beats SFT by 10+ points: SFT models learn the *form* of retrieval queries but not the *judgment* of when to retrieve—often retrieving prematurely or redundantly. RL teaches strategic search use.

    4. Key Findings: Fighting Reward Hacking in RL Training

    | Problem | Symptom | Solution | |:---|:---|:---| | Fake documents | Model fabricates <|begin_of_documents|> without searching | Strict format penalties | | Gibberish output | Base model degenerates late in training | KL divergence constraint | | Skipping retrieval | Model answers directly without searching | Two-stage training (forced retrieval in Stage 1) | | Answer stuffing | CEM reward leads to verbose answers | Switch to F1 reward |

    Additional findings:

  • Data difficulty matters: including hard training data leads to longer generations, more retrieval calls, and higher accuracy (60.8% vs 58.8% Avg CEM).
  • Data diversity drives generalization: mixing HotpotQA and 2Wiki beats single-dataset training by +10.9% Avg CEM—the model learns the *search strategy itself*, not dataset-specific patterns.

5. Limitations

1. Retriever quality bottleneck: the model's ceiling is limited by the search system's quality. 2. Multi-round retrieval coordination: currently capped at 8 retrieval rounds; harder questions may need more retrieval-reasoning loops. 3. When to search: the "uncertainty threshold" for retrieval may not be optimal; finer confidence estimation could help. 4. Beyond search: can the framework extend to code execution, calculators, databases, and broader tool ecosystems?

Still, R1-Searcher demonstrates a simple, deep point: the smartest model is not the one that knows the most, but the one that best knows what it doesn't know.

Paper Details

| Item | Content | |:---|:---| | Title | R1-Searcher: Incentivizing the Search Capability in LLMs via Reinforcement Learning | | Authors | Huatong Song, Jinhao Jiang, Yingqian Min, Jie Chen, Zhipeng Chen, Wayne Xin Zhao, Lei Fang, Ji-Rong Wen | | Institutions | Renmin University of China, DataCanvas Alaya NeW | | arXiv ID | 2503.05592 | | Date | 2025-03-07 | | Key contributions | Two-stage outcome-based RL; autonomous retrieval calls; pure RL without distillation/cold start; internalized search capability | | Key results | Llama-3.1-8B beats GPT-4o-mini (HotpotQA +48.2%); pure-RL Qwen-2.5-7B-Base achieves best results; 7B beats 32B Search-o1 by 11.4% | | Training data | 8,148 samples (HotpotQA + 2WikiMultiHopQA) | | Code | https://github.com/RUCAIBox/R1-Searcher |

Tags

#r1-searcher#reinforcement-learning#llm#rag#multi-hop-qa#search-agents#deepseek-r1#reward-hacking

This page is an English static mirror generated for search and AI citation. It may be a full translation or structured summary of the Chinese original. Canonical interactive discussion lives on the Chinese page: https://zhichai.net/topic/177619820