DeepScholar-Bench: A Live Benchmark and Automated Evaluation for Generative Research Synthesis (Aug 2025, arXiv)
Paper: https://arxiv.org/abs/2508.20033
Authors: Liana Patel, Negar Arabzadeh, Harshit Gupta, Ankita Sundar, Ion Stoica, Matei Zaharia, et al. (7 authors)
Category: Evaluation of Search Engines / Deep Research Agents
What is DeepScholar-Bench?
DeepScholar-Bench is a live benchmark and holistic automated evaluation framework for generative research synthesis — the task where AI systems (deep research agents, LLM-based search tools) retrieve real scientific literature from the web and generate comprehensive, well-cited research syntheses.
Key points
- Live design: Queries are sourced from recent papers, so systems cannot succeed through memorization; they must perform genuine retrieval over evolving web sources.
- Holistic evaluation: The accompanying DeepScholar-Gen automated evaluation pipeline assesses outputs along three axes: 1. Retrieval quality — coverage of relevant, high-quality sources; 2. Synthesis quality — organization, insight, and completeness of the generated overview; 3. Verifiability — correctness of citations and faithful attribution of claims to sources.
- Findings: Experiments show that even state-of-the-art deep research systems and leading LLMs achieve only moderate performance, indicating significant headroom in generative research synthesis.
- Availability: The benchmark, evaluation pipeline, and code are released publicly to enable reproducible, ongoing tracking of progress.
- DeepScholar-Bench: A Live Benchmark and Automated Evaluation for Generative Research Synthesis, arXiv, Aug 2025. https://arxiv.org/abs/2508.20033
Why it matters
Static QA and retrieval benchmarks (e.g., MS MARCO, BEIR, Natural Questions) measure point-in-time recall and ranking, but modern research agents must plan queries, iterate, verify citations, and synthesize findings. DeepScholar-Bench shifts evaluation from static nDCG-style metrics toward process-oriented metrics: task success, citation accuracy, and completeness of multi-hop reasoning chains.
Context in the field
The work sits at the intersection of information retrieval, RAG (retrieval-augmented generation), and agentic search, complementing related evaluation efforts such as ARES (automated RAG evaluation), QASA (research-anchored QA datasets), and AgentBoard (multi-turn LLM agent evaluation). Readers can follow the pipeline retrieval → ranking → generation/agentic synthesis → evaluation to locate complementary literature.