R-Search: Empowering LLM Reasoning with Search via Multi-Reward Reinforcement Learning
Paper Overview
- Title: R-Search: Empowering LLM Reasoning with Search via Multi-Reward Reinforcement Learning
- Authors / Affiliations: Qingfei Zhao, Ruobing Wang, Dingling Xu, Daren Zha, Limin Liu
- Published: 2025-06-04
- Paper: https://arxiv.org/abs/2506.04185
- Code / Data: https://github.com/QingFei1/R-Search
- Category: Agentic Search
- Dynamic decision-making: R-Search guides the LLM to dynamically decide when to retrieve and when to reason, rather than following a fixed retrieve-then-generate pipeline.
- Global evidence integration: Key evidence is integrated globally to enhance deep knowledge interaction between reasoning and search.
- Multi-reward RL training: During reinforcement learning, R-Search provides multi-stage, multi-type rewards to jointly optimize the entire reasoning-search trajectory.
- Up to 32.2% in-domain
- Up to 25.1% out-of-domain
- Original paper: R-Search: Empowering LLM Reasoning with Search via Multi-Reward Reinforcement Learning. arXiv:2506.04185
TL;DR
R-Search is a reinforcement learning framework that enables LLMs to autonomously perform multi-step reasoning with deep search interaction, learning optimal reasoning–search interaction trajectories via multi-reward signals for complex logic- and knowledge-intensive tasks.
Background and Motivation
Large language models have made notable progress in multi-step and long-chain reasoning. However, extending these capabilities to deep interaction with search engines remains non-trivial: models often fail to identify optimal reasoning-search interaction trajectories, resulting in suboptimal responses. Traditional RAG pipelines also tend to treat retrieval, ranking, and generation as separate stages, which limits multi-hop reasoning and real-time knowledge access. R-Search addresses this gap in the context of open-domain information access, enterprise knowledge retrieval, and conversational search.
Method
Experimental Results
Experiments were conducted on seven datasets. R-Search outperforms advanced RAG baselines by:
Quantitative details should be verified against the original PDF tables before citing specific numbers.
Key Takeaways for Search / Rec / Personalization
1. Architecture: Cascaded retrieve-rerank-generate remains mainstream, but the agentic paradigm treats retrieval frequency and strategy itself as learnable objects. 2. Data: High-quality instruction data and click/session logs matter; synthetic data must guard against knowledge leakage and distribution shift. 3. Evaluation: The gap between offline metrics and online satisfaction is widening; LLM-as-judge needs cross-validation with human evaluation. 4. Production: Latency, cost, explainability, and safety are hard constraints for industrial deployment — do not optimize academic benchmarks alone.
Limitations and Future Directions
Possible limitations include experiment scale constrained by GPU budgets, benchmarks that diverge from real user distributions, English-centric data limiting cross-lingual generalization, and safety risks of agent systems on the open web. Future work may explore more efficient test-time compute allocation, deeper integration with knowledge graphs and structured databases, and causal/fairness constraints for recommendation systems.
Engineering Checklist
| Check | Question | Suggestion | |-------|----------|------------| | Data | PII in training/index? Version management? | Partitioned indexes, anonymization, rollback-able embedding versions | | Latency | p99 budget? Retrieval steps? | Cascades + early stopping, hot-query caching, async reranking | | Quality | Do offline gains translate online? | Interleaving experiments, human audits, citation verification | | Safety | Poisoning/bias via open retrieval? | Source whitelists, adversarial detection, output filtering | | Cost | Token and GPU cost per query? | Small-model routing, distillation, hybrid sparse+dense retrieval |
Glossary
| Term | Meaning | |------|---------| | RAG | Retrieval-Augmented Generation | | nDCG | Normalized Discounted Cumulative Gain, a ranking-quality metric | | Agentic Search | Modeling search as sequential decision-making and tool invocation |
Original Abstract
> Large language models (LLMs) have notably progressed in multi-step and long-chain reasoning. However, extending their reasoning capabilities to encompass deep interactions with search remains a non-trivial challenge, as models often fail to identify optimal reasoning-search interaction trajectories, resulting in suboptimal responses. We propose R-Search, a novel reinforcement learning framework for Reasoning-Search integration, designed to enable LLMs to autonomously execute multi-step reasoning with deep search interaction, and learn optimal reasoning search interaction trajectories via multi-reward signals, improving response quality in complex logic- and knowledge-intensive tasks. R-Search guides the LLM to dynamically decide when to retrieve or reason, while globally integrating key evidence to enhance deep knowledge interaction between reasoning and search. During RL training, R-Search provides multi-stage, multi-type rewards to jointly optimize the reasoning-search trajectory. Experiments on seven datasets show that R-Search outperforms advanced RAG baselines by up to 32.2% (in-domain) and 25.1% (out-of-domain). The code and data are available at https://github.com/QingFei1/R-Search.