A Comprehensive Survey of Deep Research: Systems, Methodologies, and Applications

Overview

This post summarizes the arXiv survey A Comprehensive Survey of Deep Research: Systems, Methodologies, and Applications (June 2025) by Renjun Xu and Jingwen Peng. Deep Research denotes LLM-driven systems that perform multi-step, agentic research — combining retrieval, ranking, planning, generation, and tool invocation — to answer complex, open-ended queries.

Context and Motivation

Traditional search pipelines split retrieval, ranking, and generation, which struggles to meet LLM-era demands for natural-language interaction, multi-hop reasoning, and up-to-date knowledge. The survey maps this rapidly evolving intersection of search, recommendation, and LLM agents.

Timeline of Research Evolution

• 2019–2021: BERT rerankers and DPR establish neural retrieval.
• 2022–2023: RAG and FreshLLM push retrieval-generation fusion.
• 2024: Conversational/agentic search and generative recommendation (Gen-RecSys) boom.
• 2025–2026: RL-trained search agents, Deep Research systems, and GraphRAG become new growth areas.

Taxonomy

| Dimension | Subtypes | Representative Ideas | Strengths | Limitations | |---|---|---|---|---| | Modeling | Discriminative / generative retrieval | Bi-encoders, cross-encoders, DSI | Mature, scalable | Semantic drift, update cost | | LLM integration | RAG / Agent / Tool-use | Retrieval augmentation, search agents, API calls | Flexible, interpretable | Latency, error propagation | | Optimization | Relevance / diversity / freshness | Multi-objective LTR, RLHF, online learning | Business-aligned | Annotation scarcity | | Evaluation | Offline / online / human | nDCG, MRR, LLM-as-judge, A/B tests | Comparable | Diverges from real satisfaction |

Four main technical lines are contrasted:

• Dense retrieval: high recall, low latency; good for first-stage retrieval.
• Late interaction (e.g., ColBERT): higher accuracy but larger indexes.
• Generative IR: directly "generates" documents/ids, simplifying cascades.
• Agentic search: models search as sequential decision-making, enabling multi-hop reasoning and self-reflection.

Benchmarks and Evaluation

• Datasets: MS MARCO, BEIR, Natural Questions, domain corpora, public recommendation sets.
• Metrics: nDCG@10, MRR, Recall@k, Hit@k, human preference, task success rate, latency, token cost.
• Baselines: BM25, dense retrieval, cross-encoder rerankers, no-retrieval LLMs, commercial search APIs.
• Evaluation is shifting from static nDCG toward process metrics: task success rate, citation accuracy, and multi-hop reasoning chain completeness.

Key Insights

1. Architecture: cascade retrieve-rerank-generate remains mainstream, but agentic paradigms make retrieval count and strategy themselves learnable. 2. Data: high-quality instruction data and click/session logs are both critical; synthetic data risks leakage and distribution shift. 3. Evaluation: offline-online gaps widen; LLM-as-judge needs cross-validation with human assessment. 4. Deployment: latency, cost, interpretability, and safety are hard production constraints.

Open Problems

• Lack of unified benchmarks and reproducibility of private data
• LLM evaluator bias
• Hallucination, safety, and cost constraints for agentic systems
• Cross-lingual and multimodal generalization

Engineering Checklist

| Item | Question | Recommendation | |---|---|---| | Data | PII in training/index? Versioning? | Sharded indexes, sanitization, rollback-able embeddings | | Latency | p99 budget? Retrieval steps? | Cascades with early stop, query caching, async rerank | | Quality | Does offline gain transfer online? | Interleaving experiments, human audits, citation checks | | Security | Poisoning/bias from open retrieval? | Source whitelists, adversarial detection, output filtering | | Cost | Token and GPU cost per query? | Small-model routing, distillation, hybrid sparse-dense |

Related Entries

• A Survey of LLM-based Deep Search Agents (arXiv:2508.05668)
• A Survey of Scientific Large Language Models (arXiv:2508.21148)
• Towards Scientific Intelligence: LLM-based Scientific Agents (arXiv:2503.24047)
• Agentic Reasoning framework (arXiv:2502.04644)

Reference

• Original survey: <https://arxiv.org/abs/2506.12594>