ResearchRubrics: A Benchmark of Prompts and Rubrics for Evaluating Deep Research Agents

• arXiv: https://www.arxiv.org/abs/2511.07685
• Date: November 2025
• Authors: Manasi Sharma, Chen Bo Calvin Zhang, Chaithanya Bandi, Clinton Wang, Ankit Aich, Huy Nghiem, et al. (16 authors total)
• Category: Deep Research

Overview

Deep research agents autonomously browse the web, synthesize multiple sources, and generate long-form research reports. Evaluating such open-ended outputs is hard: traditional metrics like nDCG or exact-match do not apply, and naive LLM-as-judge scoring can be noisy or gameable. ResearchRubrics addresses this by providing a benchmark of research prompts paired with detailed, human-written rubrics that decompose report quality into checkable criteria.

Key points

• Rubric-based evaluation: each prompt is accompanied by structured criteria covering factual accuracy, citation correctness, completeness, and reasoning quality, rather than a single holistic score.
• Deep research focus: the benchmark targets agents that perform iterative search, tool use, and multi-source synthesis, not just single-turn question answering.
• Reproducibility: explicit rubrics make agent comparisons more reliable and auditable than free-form judge prompts.
• Positioning: the work sits at the intersection of agentic search, retrieval-augmented generation (RAG), and LLM evaluation methodology.

Context in the literature

The post cross-references related entries, including:

• A Comprehensive Survey of Deep Research: Systems, Methodologies, and Applications (arXiv:2506.12594)
• A Survey of LLM-based Deep Search Agents (arXiv:2508.05668)
• A Survey of Scientific Large Language Models (arXiv:2508.21148)
• Towards Scientific Intelligence: LLM-based Scientific Agents (arXiv:2503.24047)
• Agentic Reasoning (arXiv:2502.04644)

Practical implications

For teams building research or search agents:

1. Evaluation design: move beyond single scores; rubric decomposition surfaces failure modes (wrong citations, missing subtopics, unsupported claims). 2. Cost/latency: iterative deep research pipelines trade inference budget for quality; benchmarks like this help quantify that trade-off. 3. Safety: open-web retrieval introduces poisoning and bias risks; source verification and output filtering remain necessary.

> Note: Quantitative results should be verified against the original PDF; this summary is based on the paper's metadata and abstract.

Reference

• Original paper: ResearchRubrics: A Benchmark of Prompts and Rubrics For Evaluating Deep Research Agents. arXiv:2511.07685, November 2025.