ReSearch: Learning to Reason with Search for LLMs via Reinforcement Learning
- Paper: https://arxiv.org/abs/2503.19470
- Published: 2025-03-25
- Authors: Mingyang Chen, Linzhuang Sun, Tianpeng Li, Haoze Sun, Yijie Zhou, Chenzheng Zhu, et al. (13 authors)
- Motivation: Combining step-by-step reasoning with external search is hard for multi-hop questions that require multiple retrieval rounds; prior approaches often rely on supervised reasoning traces.
- Method: ReSearch embeds search operations directly into the reasoning chain. Text-based thinking controls when and how to issue searches, and retrieved results feed into further reasoning. Training uses reinforcement learning only — no supervised data on reasoning steps.
- Models: Trained on Qwen2.5-7B(-Instruct) and Qwen2.5-32B(-Instruct).
- Results: Despite training on a single dataset, the models generalize strongly across various benchmarks. RL training naturally elicits reflection and self-correction behaviors.
- ReSearch sits at the intersection of information retrieval and agentic LLM systems, where the number and strategy of retrieval steps become learnable rather than fixed.
- It relates to a lineage of work: BM25 → dense retrieval → cross-encoder reranking → generative retrieval → LLM agents with iterative, verifiable retrieval (RAG / Agentic Search).
- Closest neighbors: Search-R1 (RL for search-augmented reasoning), IRCoT (interleaving retrieval with chain-of-thought), FreshLLMs (search-engine-augmented LLMs), and Gorilla (LLM + massive APIs).
- Latency & cost: Iterative retrieval multiplies query count; cascades, early stopping, and caching matter for production.
- Evaluation: Task success, citation accuracy, and multi-hop chain completeness matter more than static nDCG for agentic systems.
- Safety: Open-web retrieval introduces poisoning and bias risks; source allowlisting and output filtering are recommended.
- Generalization: Strong cross-benchmark transfer from one training dataset suggests RL may reduce the need for curated multi-dataset supervision, but quantitative results should be verified against the original paper's tables.
- Search-R1: Training LLMs to Reason and Leverage Search Engines with RL (arXiv:2503.09516)
- Interleaving Retrieval with Chain-of-Thought Reasoning (arXiv:2212.10509)
- FreshLLMs: Refreshing LLMs with Search Engine Augmentation (arXiv:2310.03214)
- Enhancing Noise Robustness of Retrieval-Augmented LMs (arXiv:2405.20978)
- Gorilla: LLM Connected with Massive APIs (arXiv:2305.15334)
- Chen et al., "ReSearch: Learning to Reason with Search for LLMs via Reinforcement Learning," arXiv:2503.19470, 2025.
Summary
> Large Language Models (LLMs) have shown remarkable capabilities in reasoning, exemplified by the success of OpenAI-o1 and DeepSeek-R1. However, integrating reasoning with external search processes remains challenging, especially for complex multi-hop questions requiring multiple retrieval steps. We propose ReSearch, a novel framework that trains LLMs to Reason with Search via reinforcement learning without using any supervised data on reasoning steps. Our approach treats search operations as integral components of the reasoning chain, where when and how to perform searches is guided by text-based thinking, and search results subsequently influence further reasoning. We train ReSearch on Qwen2.5-7B(-Instruct) and Qwen2.5-32B(-Instruct) models and conduct extensive experiments. Despite being trained on only one dataset, our models demonstrate strong generalizability across various benchmarks. Analysis reveals that ReSearch naturally elicits advanced reasoning capabilities such as reflection and self-correction during the reinforcement learning process.