R-Search: Empowering LLM Reasoning with Search via Multi-Reward Reinforcement Learning
| Field | Detail | |-------|--------| | Title | R-Search: Empowering LLM Reasoning with Search via Multi-Reward Reinforcement Learning | | Authors | Qingfei Zhao, Ruobing Wang, Dingling Xu, Daren Zha, Limin Liu | | Published | 2025-06-04 | | Source | https://arxiv.org/abs/2506.04185 | | Type | Academic paper (Agentic Search) | | Code | https://github.com/QingFei1/R-Search |
Key points
- Problem: LLMs have advanced in multi-step, long-chain reasoning, but extending reasoning to deep interaction with search is non-trivial. Models often fail to identify optimal reasoning-search interaction trajectories, producing suboptimal answers on complex, knowledge-intensive tasks.
- Approach: R-Search is a reinforcement learning framework for Reasoning-Search integration. It lets the LLM dynamically decide when to retrieve vs. reason, while globally integrating key evidence to strengthen knowledge interaction between reasoning and search.
- Training: During RL, R-Search supplies multi-stage, multi-type rewards that jointly optimize the entire reasoning-search trajectory rather than only final answers.
- Results: Evaluated on seven datasets, R-Search outperforms advanced RAG baselines by up to 32.2% (in-domain) and 25.1% (out-of-domain).
- Architecture: Retrieval policy and iteration count can be learned via RL rather than hard-coded heuristics.
- Evaluation: Beyond nDCG-style static metrics, process indicators (multi-hop chain completeness, citation accuracy, task success rate) matter for agentic systems.
- Deployment: Watch latency budgets (number of retrieval steps, early stopping), caching, and safety concerns such as retrieval poisoning when opening search to agents.
- AceSearcher: Bootstrapping Reasoning and Search for LLMs via RL
- Synergizing RAG and Reasoning: A Systematic Review (arXiv:2504.15909)
- Agentic Information Retrieval (arXiv:2410.09713)
- Retrieval Augmented Generation and Understanding in Vision: A Survey (arXiv:2503.18016)
Original abstract
> Large language models (LLMs) have notably progressed in multi-step and long-chain reasoning. However, extending their reasoning capabilities to encompass deep interactions with search remains a non-trivial challenge, as models often fail to identify optimal reasoning-search interaction trajectories, resulting in suboptimal responses. We propose R-Search, a novel reinforcement learning framework for Reasoning-Search integration, designed to enable LLMs to autonomously execute multi-step reasoning with deep search interaction, and learn optimal reasoning search interaction trajectories via multi-reward signals, improving response quality in complex logic- and knowledge-intensive tasks. R-Search guides the LLM to dynamically decide when to retrieve or reason, while globally integrating key evidence to enhance deep knowledge interaction between reasoning and search. During RL training, R-Search provides multi-stage, multi-type rewards to jointly optimize the reasoning-search trajectory. Experiments on seven datasets show that R-Search outperforms advanced RAG baselines by up to 32.2% (in-domain) and 25.1% (out-of-domain). The code and data are available at https://github.com/QingFei1/R-Search.
Context and significance
R-Search sits at the intersection of RAG and agentic search. Classic pipelines treat retrieval, ranking, and generation as fixed stages; the new variable in the LLM era is the action space itself — whether to retrieve, how many times, and which tools to call. By making the reasoning-search trajectory a learnable object optimized with process-level rewards, R-Search exemplifies the shift from one-shot retrieval toward iterative, verifiable, planning-driven knowledge access.
Takeaways for practitioners
Limitations to verify against the paper
Experiment scale and compute budget, benchmark-vs-real-user distribution gaps, and cross-lingual generalization should be checked in the full PDF before citing quantitative results.