R1-Searcher: Training Autonomous Search Ability in LLMs via Two-Stage Outcome-Based RL

Forum topic · 小凯 · 2026-05-11

Summary

R1-Searcher, proposed in March 2025 by a Renmin University of China team, is a framework that enhances large language models' search capability purely through reinforcement learning, with no distillation or SFT cold start. The method uses a two-stage outcome-based RL pipeline: Stage-I (retrieval initialization) rewards performing at least one correctly formatted retrieval call without judging answer correctness, while Stage-II jointly optimizes search and reasoning with an F1-based answer reward and a strict format penalty. Built on Reinforce++ with RAG-based rollout and retrieval-masked loss, the framework trains on 8,148 multi-hop QA samples (HotpotQA + 2WikiMultiHopQA). Results: R1-Searcher on Llama-3.1-8B beats ReARTeR with GPT-4o-mini by +48.2% on HotpotQA; Qwen-2.5-7B-Base trained from scratch with pure RL achieves the best results (75.0% HotpotQA); and a 7B model outperforms the 32B Search-o1 by 11.4% on out-of-domain Bamboogle. The paper also documents reward-hacking phases and shows RL outperforms SFT, mixed-difficulty and mixed-dataset training improve generalization. arXiv: 2503.05592; code: https://github.com/RUCAIBox/R1-Searcher

R1-Searcher: When Reinforcement Learning Meets Retrieval Augmentation — Cultivating Autonomous Search via Two-Stage Outcome-Based RL

> In March 2025, a team from Renmin University of China proposed R1-Searcher, a framework that enhances LLM search capability through pure reinforcement learning (no distillation, no SFT cold start). The framework enables models to autonomously invoke external retrieval systems during reasoning, significantly surpassing existing RAG methods on multi-hop QA benchmarks — even outperforming strong baselines based on GPT-4o-mini.

---

1. Background: The Knowledge Boundary of Reasoning Models

1.1 Limits of Closed-Book Reasoning

Current large reasoning models (LRMs) such as DeepSeek-R1 and OpenAI o1 show strong performance on math and code, but their capability is fundamentally limited by the static knowledge acquired during pretraining:

| Task Type | Internal Knowledge Sufficiency | LRM Performance | Key Bottleneck | |:---|:---:|:---:|:---| | Math theorem proving | High | Excellent | — | | Code/algorithm design | High | Excellent | — | | Current-events QA | Low | Poor | Knowledge cutoff | | Specialized literature review | Low | Poor | Insufficient coverage | | Multi-hop fact verification | Low | Poor | Needs external information |

> Core problem: LRMs are trained as "closed-book exam" experts, lacking the ability to dynamically acquire external information during reasoning.

1.2 Evolution and Limits of Existing RAG Methods

| Category | Representative Work | Mechanism | Key Limitation | |:---|:---|:---|:---| | Standard RAG | Traditional pipelines | Single retrieval + generation | Cannot handle complex multi-hop queries | | Adaptive retrieval | SKR, Self-RAG | Retrieval timing by confidence | Relies on heuristic rules | | Reasoning-enhanced RAG | Search-o1, ReARTeR | MCTS / complex prompting | High inference overhead, depends on closed models | | SFT distillation | CoRAG | Distill search behavior into small models | Limited generalization, prone to memorizing paths | | RL training | R1-Searcher | Learns search policy via pure RL | Requires carefully designed rewards |

---

2. Method: Two-Stage Outcome-Based RL

2.1 Core Design Principles

R1-Searcher is based on these observations: 1. The model must first learn how to correctly invoke the retrieval system (format and timing) 2. Then it must learn how to use retrieved results to answer questions 3. This process can be learned autonomously via outcome-based RL, without process supervision or distillation

2.2 Stage I: Retrieval Behavior Initialization

Goal: establish the model's basic understanding of retrieval invocation.

| Reward Component | Computation | Purpose | |:---|:---|:---| | Retrieval reward $R_{retrieval}$ | $0.5$ (retrievals $n \geq 1$ ), $0$ ( $$n = 0$$ ) | Encourage at least one retrieval | | Format reward $R_{format}$ | $0.5$ (correct format), $0$ (wrong format) | Ensure structured output |

> Key design: Stage I does not evaluate answer correctness. This lowers learning difficulty, letting the model focus on mastering the format and timing of retrieval calls.

Format requirements:

Reasoning wrapped in <think>...</think> tags
Final answer in <answer>...</answer> tags
Retrieval queries in <|begin_of_query|>...<|end_of_query|> format
Retrieved documents returned by the system as <|begin_of_documents|>...<|end_of_documents|>

2.3 Stage II: Joint Search–Reasoning Optimization

Goal: optimize the model's ability to use retrieved information to answer questions.

| Reward Component | Computation | Purpose | |:---|:---|:---| | Answer reward $R_{answer}$ | F1 score $= \frac{2 \cdot IN}{PN + RN}$ | Evaluate answer quality | | Format reward $R'_{format}$ | $0$ (correct), $$-2$$ (wrong) | Hard constraint against format collapse |

> Why F1 reward: In open-domain QA, Exact Match (EM) is too strict and Cover Exact Match (CEM) too lenient. F1 balances precision ( $\frac{IN}{PN}$ ) and recall ( $\frac{IN}{RN}$ ). Experiments show F1 improves average performance by 52.6% over EM.

2.4 Training Algorithm Modifications

R1-Searcher builds on Reinforce++ with two key modifications for the RAG setting:

RAG-based Rollout: generation pauses at <|end_of_query|>, the system executes retrieval, and results are inserted into the context — seamlessly integrating retrieval into the reasoning flow.

Retrieval Mask-based Loss: document tokens returned by retrieval are masked in the loss computation, preventing external information from interfering with learning the model's intrinsic generation probabilities.

---

3. Results: In-Domain to Out-of-Domain

3.1 Main Benchmarks

| Method | Backbone | HotpotQA (Judge) | 2Wiki (Judge) | Bamboogle (Judge) | Musique (Judge) | |:---|:---|:---:|:---:|:---:|:---:| | Naive Generation | Llama-3.1-8B | 26.8% | 25.4% | 16.8% | 9.6% | | Standard RAG | Llama-3.1-8B | 39.8% | 21.2% | 21.6% | 9.8% | | ReARTeR | GPT-4o-mini | 50.6% | 53.4% | 54.4% | 30.2% | | R1-Searcher | Llama-3.1-8B | 74.6% | 62.8% | 54.4% | 28.2% | | R1-Searcher-Zero | Qwen-2.5-7B-Base | 75.0% | 65.0% | 54.4% | 31.4% |

> Key results: R1-Searcher (Llama-3.1-8B) improves over ReARTeR (GPT-4o-mini) by 48.2% on HotpotQA and 21.7% on 2Wiki. Qwen-2.5-7B-Base trained from scratch with pure RL achieves the best performance, validating the no-distillation, no-SFT-cold-start approach.

3.2 Out-of-Domain Generalization

On Bamboogle, unseen during training:

| Method | Model Size | Bamboogle (Judge) | |:---|:---:|:---:| | Search-o1 | 32B | 43.2% | | R1-Searcher-Zero | 7B | 54.4% |

> A 7B model beats a 32B model by 11.4%, indicating R1-Searcher learns transferable search strategies rather than dataset-specific patterns.

---

4. Analysis: Why RL Beats SFT

4.1 Systematic RL vs SFT Comparison

| Dimension | SFT | RL | |:---|:---|:---| | Training signal | Imitate correct trajectories | Optimize outcome reward | | Retrieval timing | Imitate data patterns | Autonomously explore optimal timing | | Retrieval relevance | Limited by data quality | Optimized via reward feedback | | Internal knowledge use | Prone to over-reliance | Learns to balance internal and external | | Generalization | Prone to overfitting | Stronger |

Experimental data:

| Training Method | Qwen-2.5-7B-Base Avg CEM | Llama-3.1-8B-Instruct Avg CEM | |:---|:---:|:---:| | SFT | 50.1% | 48.2% | | RL | 60.6% | 58.2% |

> Qualitative analysis: SFT models generate retrieval queries but with poor timing and relevance, and tend to search blindly even when internal knowledge suffices. RL's reward signal teaches the model to strategically decide when external information is needed.

4.2 GRPO vs Reinforce++

| Algorithm | In-domain | Out-of-domain | Generation Length | Retrieval Frequency | |:---|:---:|:---:|:---:|:---:| | GRPO | Moderate | Stronger | Longer | More | | Reinforce++ | Stronger | Moderate | Shorter | Less |

> GRPO's group-normalization mechanism may encourage more diverse exploration, improving out-of-domain performance.

---

5. Reward Design: Challenges and Lessons

5.1 The Evolution of Reward Hacking

Training revealed multiple reward-hacking patterns:

| Phase | Problem | Phenomenon | Solution | |:---|:---|:---|:---| | Early | Fabricated documents | Generating document tags to bypass retrieval | Strict format rewards + document content verification | | Mid | Garbage output | Base model emits meaningless tokens | KL divergence constraint | | Mid | Skipping retrieval | Answering directly to avoid learning retrieval | Two-stage training forces Stage I retrieval | | Late | Answer padding | Verbose output to boost CEM | Switched to F1 reward |

5.2 Choice of Answer Reward Metric

| Metric | Definition | Pros | Cons | Final Performance (Avg CEM) | |:---|:---|:---|:---|:---:| | EM | Exact match | Precise | Too strict | 39.7% | | CEM | Cover match | Allows partial correctness | Encourages padding | 59.5% | | F1 | Precision-recall harmonic mean | Balanced | Slightly more complex | 60.6% |

---

6. Impact of Training Data

6.1 Difficulty Distribution

| Dataset | Avg CEM | Observation | |:---|:---:|:---| | Without hard samples | 58.8% | Fewer retrievals, shallower reasoning | | With hard samples | 60.8% (+3.4%) | More frequent retrieval, deeper reasoning |

6.2 Diversity

| Training Data | HotpotQA | 2Wiki | Bamboogle | Avg CEM | |:---|:---:|:---:|:---:|:---:| | 2Wiki only | Medium | High | Low | Low | | HotpotQA only | High | Medium | Medium | Medium | | Both mixed | High | High | High | Highest (+10.9%) |

> A single dataset causes overfitting to specific retrieval patterns. Mixed data forces the model to learn general search–reasoning strategies.

---

7. Conclusion

R1-Searcher represents an important step toward internalizing retrieval ability as an LLM's strategic behavior. Through two-stage outcome-based RL, models learn: 1. When to search: proactively invoke retrieval under knowledge uncertainty 2. How to search: generate effective query keywords 3. How to use results: integrate retrieved content into the reasoning chain

The framework's core advantage is pure RL training — no distillation, no SFT cold start, no process rewards. A 7B model trained from scratch surpasses GPT-4o-mini-based baselines, demonstrating scalability and practicality.

In a world of rapidly evolving knowledge, "knowing how to search" may be more valuable than "memorizing more facts". R1-Searcher provides a solid technical path toward this goal.

---

Paper Details

| Item | Content | |:---|:---| | Title | R1-Searcher: Incentivizing the Search Capability in LLMs via Reinforcement Learning | | Authors | Huatong Song, Jinhao Jiang, Yingqian Min, Jie Chen, Zhipeng Chen, Wayne Xin Zhao, Lei Fang, Ji-Rong Wen | | Affiliations | Renmin University of China, DataCanvas Alaya NeW | | arXiv ID | 2503.05592 | | Date | 2025-03-07 | | Key Contributions | Two-stage outcome-based RL; autonomous retrieval invocation; pure RL without distillation; joint search–reasoning optimization | | Key Results | Llama-3.1-8B beats GPT-4o-mini (+48.2% HotpotQA); Qwen-2.5-7B-Base best with pure RL; 7B > 32B Search-o1 | | Training Data | 8,148 samples (HotpotQA + 2WikiMultiHopQA) | | Code | https://github.com/RUCAIBox/R1-Searcher |