R-Search: Empowering LLM Reasoning with Search via Multi-Reward Reinforcement Learning
- Paper: arXiv:2506.04185
- Authors: Qingfei Zhao, Ruobing Wang, Dingling Xu, Daren Zha, Limin Liu
- Date: June 4, 2025
- Category: Agentic Search
- Code: https://github.com/QingFei1/R-Search
- Dynamic retrieval policy: The LLM learns when to retrieve versus when to reason, rather than following a fixed retrieve-then-generate pipeline.
- Global evidence integration: Key evidence is integrated globally to deepen knowledge interaction between reasoning and search steps.
- Multi-stage, multi-type rewards: RL training jointly optimizes the full reasoning-search trajectory rather than only the final answer.
- Strong results: Up to 32.2% (in-domain) and 25.1% (out-of-domain) improvement over advanced RAG baselines across seven datasets.
- Open source: Code and data are publicly available for reproduction.
- A Systematic Framework for Enterprise Knowledge Retrieval (arXiv:2512.05411)
- Retrieval Augmented Generation and Understanding in Vision: A Survey (arXiv:2503.18016)
- Synergizing RAG and Reasoning: A Systematic Review (arXiv:2504.15909)
- AceSearcher: Bootstrapping Reasoning and Search for LLMs via Reinforcement Learning (arXiv:2509.24193)
- Agentic Information Retrieval (arXiv:2410.09713)
- Latency: p99 budgets should drive retrieval step counts; use cascades with early stopping, query caching, and async reranking.
- Quality: Verify that offline gains translate to online satisfaction via interleaving experiments and human audits with citation checking.
- Safety: Open retrieval can introduce poisoning or bias; consider source whitelisting, adversarial detection, and output filtering.
- Cost: Track per-query token and GPU usage; consider model routing, distillation, and hybrid sparse-dense retrieval.
TL;DR
R-Search is a reinforcement learning framework for Reasoning-Search integration. It enables LLMs to autonomously execute multi-step reasoning with deep search interaction, learning optimal reasoning-search interaction trajectories via multi-reward signals.
Original Abstract
> Large language models (LLMs) have notably progressed in multi-step and long-chain reasoning. However, extending their reasoning capabilities to encompass deep interactions with search remains a non-trivial challenge, as models often fail to identify optimal reasoning-search interaction trajectories, resulting in suboptimal responses. We propose R-Search, a novel reinforcement learning framework for Reasoning-Search integration, designed to enable LLMs to autonomously execute multi-step reasoning with deep search interaction, and learn optimal reasoning search interaction trajectories via multi-reward signals, improving response quality in complex logic- and knowledge-intensive tasks. R-Search guides the LLM to dynamically decide when to retrieve or reason, while globally integrating key evidence to enhance deep knowledge interaction between reasoning and search. During RL training, R-Search provides multi-stage, multi-type rewards to jointly optimize the reasoning-search trajectory. Experiments on seven datasets show that R-Search outperforms advanced RAG baselines by up to 32.2% (in-domain) and 25.1% (out-of-domain). The code and data are available at https://github.com/QingFei1/R-Search.
Key points
Context and positioning
The paper sits at the intersection of RAG and agentic search. Classical search stacks treat recall, ranking, and generation as separate funnel stages; the LLM era introduces two new variables — the reasoning budget and the action space (whether to retrieve, how many times, and which tools to call). Agentic approaches like R-Search make the retrieval count and strategy themselves learnable objects.
Typical pipeline components in this problem space include retrievers, rerankers, planners, memory modules, and tool interfaces, trained via supervised fine-tuning, contrastive learning, distillation, or RL (including process rewards), with inference strategies ranging from single-shot retrieval to iterative retrieval, parallel sub-queries, and budget-controlled early stopping.