ReSearch: Learning to Reason with Search for LLMs via Reinforcement Learning

Paper: https://arxiv.org/abs/2503.19470 (March 25, 2025)

Authors: Mingyang Chen, Linzhuang Sun, Tianpeng Li, Haoze Sun, Yijie Zhou, Chenzheng Zhu, et al. (13 authors total)

Overview

Large Language Models (LLMs) have shown remarkable capabilities in reasoning, exemplified by the success of OpenAI-o1 and DeepSeek-R1. However, integrating reasoning with external search processes remains challenging, especially for complex multi-hop questions requiring multiple retrieval steps. ReSearch is a novel framework that trains LLMs to reason with search via reinforcement learning without using any supervised data on reasoning steps.

Key Ideas

• Search as part of the reasoning chain: Search operations are treated as integral components of reasoning. When and how to perform searches is guided by text-based thinking, and search results subsequently influence further reasoning.
• RL-only training: No supervised demonstrations of reasoning or retrieval steps are needed; the model learns to invoke search through reinforcement learning alone.
• Models: ReSearch is trained on Qwen2.5-7B(-Instruct) and Qwen2.5-32B(-Instruct).
• Generalization: Despite training on only one dataset, the models demonstrate strong generalizability across various benchmarks.
• Emergent capabilities: Analysis reveals that ReSearch naturally elicits advanced reasoning capabilities such as reflection and self-correction during the RL process.

Context in the Field

• The work sits at the intersection of information retrieval and agentic LLM systems, addressing how to redistribute the responsibilities of retrieval, ranking, generation, and tool calling in the LLM era.
• It extends RAG from a one-shot retrieval step into an iterative, plannable process, moving evaluation from static metrics like nDCG toward task success rate and multi-hop reasoning chain completeness.
• Closely related efforts include Search-R1 (training LLMs to leverage search engines with RL), IRCoT (interleaving retrieval with chain-of-thought reasoning), and FreshLLMs (search-engine augmentation).

Engineering Considerations (from the forum discussion)

• Latency & cost: iterative retrieval adds steps; cascades with early stopping, query caching, and budget control help meet p99 targets.
• Quality: offline gains should be validated with interleaved online experiments and citation checks.
• Safety: open-ended retrieval can introduce poisoning or bias; consider source whitelisting and output filtering.

References

• ReSearch: Learning to Reason with Search for LLMs via Reinforcement Learning — arXiv:2503.19470