Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning

Overview

• Authors: Bowen Jin, Hansi Zeng, Zhenrui Yue, Jinsung Yoon, Sercan Arik, Dong Wang, et al. (8 authors)
• Published: 2025-03-12
• Paper: https://arxiv.org/abs/2503.09516
• Code: https://github.com/PeterGriffinJin/Search-R1

What the paper proposes

Efficiently acquiring external knowledge and up-to-date information is essential for effective reasoning and text generation in large language models (LLMs). Prompting advanced LLMs with reasoning capabilities to use search engines during inference is often suboptimal, as the LLM might not fully possess the capability on how to interact optimally with the search engine.

Search-R1 extends reinforcement learning (RL) for reasoning frameworks so that the LLM learns to autonomously generate (multiple) search queries during step-by-step reasoning with real-time retrieval. It optimizes LLM reasoning trajectories with multi-turn search interactions, using:

• Retrieved token masking for stable RL training;
• A simple outcome-based reward function.

Results

Experiments on seven question-answering datasets show that Search-R1 improves performance by:

• 41% (Qwen2.5-7B)
• 20% (Qwen2.5-3B)

over various RAG baselines under the same setting. The paper further provides empirical insights into RL optimization methods, LLM choices, and response length dynamics in retrieval-augmented reasoning.

Key points and context

• Search-R1 exemplifies the agentic search paradigm: instead of a one-shot retrieve-then-generate pipeline, the number and strategy of retrieval steps become learnable decisions within the RL framework.
• Compared to classical search stacks (recall → re-rank → generation), LLM-era systems must additionally allocate a reasoning budget and an action space (whether to retrieve, how many times, which tools to call).
• Evaluation is shifting from static metrics like nDCG toward task success rate, citation accuracy, and multi-hop reasoning chain completeness.

Limitations to keep in mind

• Experimental scale may be constrained by GPU budgets; benchmark distributions may differ from real user traffic.
• Open-web retrieval introduces risks of poisoning, bias, and hallucination; production deployment needs source whitelisting and output filtering.
• Latency, cost, and interpretability remain hard constraints for industrial adoption beyond academic benchmarks.

Abstract (original)

> Efficiently acquiring external knowledge and up-to-date information is essential for effective reasoning and text generation in large language models (LLMs). Prompting advanced LLMs with reasoning capabilities to use search engines during inference is often suboptimal, as the LLM might not fully possess the capability on how to interact optimally with the search engine. This paper introduces Search-R1, an extension of reinforcement learning (RL) for reasoning frameworks where the LLM learns to autonomously generate (multiple) search queries during step-by-step reasoning with real-time retrieval. Search-R1 optimizes LLM reasoning trajectories with multi-turn search interactions, leveraging retrieved token masking for stable RL training and a simple outcome-based reward function. Experiments on seven question-answering datasets show that Search-R1 improves performance by 41% (Qwen2.5-7B) and 20% (Qwen2.5-3B) over various RAG baselines under the same setting. This paper further provides empirical insights into RL optimization methods, LLM choices, and response length dynamics in retrieval-augmented reasoning. The code and model checkpoints are available at https://github.com/PeterGriffinJin/Search-R1.

Related entries

• Enhancing Noise Robustness of Retrieval-Augmented Language Models (arXiv:2405.20978)
• FreshLLMs: Refreshing Large Language Models with Search Engine Augmentation (arXiv:2310.03214)
• Gorilla: Large Language Model Connected with Massive APIs (arXiv:2305.15334)
• Interleaving Retrieval with Chain-of-Thought Reasoning (arXiv:2212.10509)
• Long-form factuality in large language models (arXiv:2403.18802)
• ReSearch: Learning to Reason with Search for LLMs via Reinforcement Learning (arXiv:2503.19470)