DeepScholar-Bench: A Live Benchmark and Automated Evaluation for Generative Research Synthesis
> Source: arXiv:2508.20033, August 2025. Listed in the "Evaluation of Search Engines" section of this collection.
Overview
DeepScholar-Bench is a live benchmark and automated evaluation framework for generative research synthesis — the task of producing comprehensive, well-cited research summaries from the scientific literature, a core capability of modern "deep research" agents.
- Authors: Liana Patel, Negar Arabzadeh, Harshit Gupta, Ankita Sundar, Ion Stoica, Matei Zaharia, et al. (7 authors total)
- Link: https://arxiv.org/abs/2508.20033
- Published: August 2025, arXiv
- Evaluation of Retrieval-Augmented Generation: A Survey
- ARES: An Automated Evaluation Framework for RAG
- AI Search Has A Citation Problem (CJR, Mar 2025)
- A Dataset of Information-Seeking Questions and Answers Anchored in Research Papers
- When assessing deep research systems, evaluate citation fidelity, not just fluency or coverage.
- Prefer live or recently-refreshed evaluation sets to avoid contamination artifacts.
- Cross-validate LLM-as-judge scores with human audits, especially for citation claims.
- In production, weigh latency, cost, and safety constraints alongside benchmark scores.
Why a Live Benchmark?
Static benchmarks for research synthesis face two recurring problems:
1. Data contamination — models may have seen the queries, reference reports, or underlying papers during pretraining or fine-tuning. 2. Saturation — fixed test sets are quickly exhausted, making it hard to measure frontier progress.
A *live* design grounds evaluation in recently published research, and an *automated* evaluator (DeepScholar-REF) reduces the cost of human grading while supporting systematic comparison of systems.
Positioning in the Evaluation Literature
This work belongs to a growing line of research on evaluating retrieval-augmented and agentic generation, including:
Evaluation of generative IR is shifting from static ranking metrics (nDCG, MRR) toward process-level measures: task success, citation accuracy and completeness, and faithfulness of multi-hop synthesis chains.
Takeaways for Practitioners
Notes
This entry is based on the paper's metadata and abstract; consult the original PDF for exact benchmark construction, metrics, and quantitative results before citing specific numbers.