A Comprehensive Survey on Reinforcement Learning-based Agentic Search: Foundations, Roles, Optimizations, Evaluations, and Applications

Overview

• Paper: A Comprehensive Survey on Reinforcement Learning-based Agentic Search: Foundations, Roles, Optimizations, Evaluations, and Applications
• Authors: Minhua Lin, Zongyu Wu, Zhichao Xu, Hui Liu, Xianfeng Tang, Qi He, et al. (10 authors in total)
• Published: 2025-10-19
• Source: https://arxiv.org/abs/2510.16724
• Repository: https://github.com/ventr1c/Awesome-RL-based-Agentic-Search-Papers
• Type: Survey

One-sentence Summary

This is the first comprehensive survey of RL-based agentic search, organizing the emerging field by the functional roles of RL, the optimization strategies used, and the scope of application.

Background and Motivation

The advent of large language models (LLMs) has transformed information access and reasoning through open-ended natural language interaction. However, LLMs remain limited by static knowledge, factual hallucinations, and the inability to retrieve real-time or domain-specific information. Retrieval-Augmented Generation (RAG) mitigates these issues by grounding model outputs in external evidence, but traditional RAG pipelines are often single-turn and heuristic, lacking adaptive control over retrieval and reasoning. Recent advances in agentic search address these limitations by enabling LLMs to plan, retrieve, and reflect through multi-step interaction with search environments. Within this paradigm, reinforcement learning (RL) offers a powerful mechanism for adaptive and self-improving search behavior.

Core Contributions

• A unified perspective that brings scattered related work into a comparable framework.
• A clear decomposition of method components (representation learning, retrievers, rerankers, planners, generators, feedback mechanisms) to support engineering adoption.
• Reproducible benchmarks, datasets, and taxonomy tables that lower the entry barrier for follow-up research.
• Discussion of interfaces with LLM tool calling, reinforcement learning, and multi-agent collaboration, including paths from research prototypes to industrial systems.
• Explicitly listed open problems: evaluation reliability, latency and cost, hallucination and safety, and cross-lingual and multimodal extension.

Taxonomy

| Dimension | Subcategories | Representative Ideas | Strengths | Limitations | |-----------|---------------|----------------------|-----------|-------------| | Modeling paradigm | Discriminative / generative retrieval | Bi-encoders, cross-encoders, DSI, GPT indexes | Mature, scalable | Semantic drift, update cost | | LLM integration | RAG / Agent / Tool-use | Retrieval augmentation, search agents, API calls | Flexible, interpretable | Latency, error propagation | | Optimization objectives | Relevance / diversity / freshness | Multi-objective LTR, RLHF, online learning | Business-aligned | Scarce annotations | | Evaluation | Offline / Online / Human | nDCG, MRR, LLM-as-judge, A/B tests | Comparable | Deviates from real satisfaction |

Positioning Among Research Lines

The survey places agentic search alongside three established lines of work in neural information retrieval:

• Dense retrieval: high recall, low latency; well suited to first-stage retrieval.
• Late interaction (e.g., ColBERT): higher accuracy but larger indexes.
• Generative IR: directly "generates" documents via tokens or docids, simplifying cascades.
• Agentic search: models search as sequential decision-making, enabling multi-hop reasoning and self-reflection.

Timeline of the field: 2019–2021, BERT rerankers and DPR laid the foundation of neural retrieval; 2022–2023, RAG and FreshLLMs drove retrieval-generation integration; from 2024, conversational/agentic search and generative recommenders (Gen-RecSys) surged; 2025–2026, RL-trained search agents, Deep Research, and GraphRAG have become new growth frontiers.

Evaluation Paradigms Covered

• Datasets: MS MARCO, BEIR, Natural Questions, domain-specific corpora, and public recommendation datasets.
• Metrics: nDCG@10, MRR, Recall@k, Hit@k, human preference, task success rate, latency, and token cost.
• Baselines: BM25, dense retrieval, cross-encoder reranking, retrieval-free LLMs, and commercial search APIs.
• Ablations: contributions of modules such as number of retrieval steps, reranking depth, and training data scale.

Key Insights

1. Architecture: cascaded retrieve–rerank–generate remains mainstream, but the agentic paradigm makes the *number and strategy of retrieval steps* themselves learnable. 2. Data: high-quality instruction data and click/session logs are equally critical; synthetic data must guard against knowledge leakage and distribution shift. 3. Evaluation: the gap between offline metrics and online satisfaction is widening; LLM-as-judge needs cross-validation with human assessment. 4. Productization: latency, cost, interpretability, and safety policies are hard constraints in industrial deployment and cannot be ignored in favor of academic benchmarks.

Open Problems and Future Directions

The authors point to insufficient unified benchmarks, non-reproducible private data, LLM evaluation bias, and safety and cost constraints of agentic systems. Future directions include finer-grained process supervision, joint retrieval-reasoning training, enterprise metadata governance, and multimodal and cross-lingual consistency.

Original Abstract (verbatim)

> The advent of large language models (LLMs) has transformed information access and reasoning through open-ended natural language interaction. However, LLMs remain limited by static knowledge, factual hallucinations, and the inability to retrieve real-time or domain-specific information. Retrieval-Augmented Generation (RAG) mitigates these issues by grounding model outputs in external evidence, but traditional RAG pipelines are often single turn and heuristic, lacking adaptive control over retrieval and reasoning. Recent advances in agentic search address these limitations by enabling LLMs to plan, retrieve, and reflect through multi-step interaction with search environments. Within this paradigm, reinforcement learning (RL) offers a powerful mechanism for adaptive and self-improving search behavior. This survey provides the first comprehensive overview of *RL-based agentic search*, organizing the emerging field along three complementary dimensions: (i) What RL is for (functional roles), (ii) How RL is used (optimization strategies), and (iii) Where RL is applied (scope of optimization). We summarize representative methods, evaluation protocols, and applications, and discuss open challenges and future directions toward building reliable and scalable RL driven agentic search systems. We hope this survey will inspire future research on the integration of RL and agentic search. Our repository is available at https://github.com/ventr1c/Awesome-RL-based-Agentic-Search-Papers.

Related Entries

• A Survey of Conversational Search (Oct 2024, arXiv 2410.15576)
• A Survey of Large Language Model Empowered Agents for Recommendation (arXiv 2503.05659)
• A Survey of Model Architectures in Information Retrieval (Jan 2025, arXiv 2502.14822)
• A Survey on AI Search with Large Language Models (July 2025)
• A Survey on Knowledge-Oriented Retrieval-Augmented Generation (arXiv 2503.10677)
• Cross-Modal Retrieval: A Systematic Review of Methods and Future Directions

Glossary

| Term | Meaning | |------|---------| | IR | Information Retrieval | | RAG | Retrieval-Augmented Generation | | LTR | Learning to Rank | | nDCG | Normalized Discounted Cumulative Gain, a ranking quality metric | | Agentic Search | Treating search as sequential decision-making and tool invocation by an agent | | Gen-IR | Generative Information Retrieval |

References

• Original paper: A Comprehensive Survey on Reinforcement Learning-based Agentic Search: Foundations, Roles, Optimizations, Evaluations, and Applications. https://arxiv.org/abs/2510.16724