DeepResearch Bench: A Comprehensive Benchmark for Deep Research Agents
arXiv: https://arxiv.org/abs/2506.11763 Authors / Affiliations: Mingxuan Du, Benfeng Xu, Chiwei Zhu, Xiaorui Wang, Zhendong Mao Type: Academic paper Section: Deep Research, Evaluation of Search Engines
Overview
DeepResearch Bench is a comprehensive benchmark for evaluating deep research agents — LLM-based systems that perform open-ended research by combining multi-step web search, planning, and long-form report generation. The paper was posted to arXiv in June 2025.
The benchmark targets two core capabilities:
1. Report generation quality — evaluating the depth, breadth, accuracy, and presentation of long-form research reports produced by agents. 2. Web search capability — assessing how effectively an agent retrieves and selects information from the live web for complex research tasks.
Evaluation Methodology
For report quality assessment, the paper introduces RACE (Reference-based Adaptive Critique-Evaluate), a framework that uses LLM-as-judge to compare generated reports against a curated set of reference documents, producing task-specific, criterion-based scores. This aims to improve the reliability of automatic evaluation compared with generic LLM scoring.
For search ability, the benchmark provides research tasks grounded in real-world information needs, requiring iterative querying, source selection, and synthesis.
Key points
- Fills the gap of standardized evaluation for deep research agents combining search and generation.
- Evaluates two axes: report quality and web search effectiveness.
- RACE framework applies reference-grounded, LLM-as-judge evaluation for long-form reports.
- Positioned in the emerging agentic search literature, where retrieval strategy itself is a learnable behavior.
- A Comprehensive Survey of Deep Research: Systems, Methodologies, and Applications
- A Survey of LLM-based Deep Search Agents
- A Survey of Scientific Large Language Models
- Towards Scientific Intelligence: A Survey of LLM-based Scientific Agents
- Agentic Reasoning: A Streamlined Framework for Enhancing LLM Reasoning (arXiv:2502.04644)
- Precise quantitative results and dataset statistics should be verified against the original PDF.
- The broader context: evaluation is shifting from static IR metrics (nDCG, MRR) toward process-level metrics such as task success, citation accuracy, and multi-hop reasoning completeness.
- Practitioners should weigh latency, cost, and safety constraints when translating benchmark gains into production deep-research systems.
- Original paper: DeepResearch Bench: A Comprehensive Benchmark for Deep Research Agents. arXiv:2506.11763, June 2025. https://arxiv.org/abs/2506.11763