Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning

Overview

• Paper: Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning (arXiv:2503.09516, March 2025)
• Authors: Bowen Jin, Hansi Zeng, Zhenrui Yue, Jinsung Yoon, Sercan Arik, Dong Wang, et al. (8 authors total)
• Code: https://github.com/PeterGriffinJin/Search-R1

Motivation

Efficiently acquiring external knowledge and up-to-date information is essential for effective reasoning and text generation in large language models (LLMs). Simply prompting advanced reasoning LLMs to use a search engine during inference is often suboptimal, because the model may not know how to interact with the search engine optimally — when to search, what to query, or how to use the results.

Method

Search-R1 extends reinforcement learning (RL) for reasoning frameworks so the LLM learns to autonomously generate (multiple) search queries during step-by-step reasoning with real-time retrieval. Key design elements:

1. Multi-turn search interactions: the model interleaves reasoning with issuing search queries and consuming retrieved content. 2. Retrieved token masking: retrieved tokens are masked in the RL objective for stable training. 3. Outcome-based reward: a simple result-level reward function optimizes full reasoning-plus-search trajectories.

Results

• Evaluated on seven question-answering datasets.
• Search-R1 improves performance by 41% (Qwen2.5-7B) and 20% (Qwen2.5-3B) over various RAG baselines under the same settings.
• The paper provides empirical insights into RL optimization methods, LLM choices, and response-length dynamics in retrieval-augmented reasoning.

Takeaways for Search / RAG Systems

• Agentic retrieval: the number and policy of retrieval calls become learnable, rather than fixed pipeline steps.
• Stable RL for tool use: masking non-generated (retrieved) tokens is important for training stability.
• Practical constraints: latency, cost, hallucination/safety, and evaluation reliability remain open challenges for deploying agentic retrieval systems in production.

Related Work

• FreshLLMs: Refreshing Large Language Models with Search Engine Augmentation
• ReSearch: Learning to Reason with Search for LLMs via Reinforcement Learning
• Interleaving Retrieval with Chain-of-Thought Reasoning
• Gorilla: Large Language Model Connected with Massive APIs

Original Abstract

> Efficiently acquiring external knowledge and up-to-date information is essential for effective reasoning and text generation in large language models (LLMs). Prompting advanced LLMs with reasoning capabilities to use search engines during inference is often suboptimal, as the LLM might not fully possess the capability on how to interact optimally with the search engine. This paper introduces Search-R1, an extension of reinforcement learning (RL) for reasoning frameworks where the LLM learns to autonomously generate (multiple) search queries during step-by-step reasoning with real-time retrieval. Search-R1 optimizes LLM reasoning trajectories with multi-turn search interactions, leveraging retrieved token masking for stable RL training and a simple outcome-based reward function. Experiments on seven question-answering datasets show that Search-R1 improves performance by 41% (Qwen2.5-7B) and 20% (Qwen2.5-3B) over various RAG baselines under the same setting. This paper further provides empirical insights into RL optimization methods, LLM choices, and response length dynamics in retrieval-augmented reasoning. The code and model checkpoints are available at https://github.com/PeterGriffinJin/Search-R1.