Mind2Web 2: Evaluating Agentic Search with Agent-as-a-Judge
Overview
Agentic search systems—such as Deep Research agents that autonomously browse the web, synthesize information, and return comprehensive citation-backed answers—represent a major shift in how users interact with web-scale information. However, the growing complexity and open-endedness of agentic search have outpaced existing evaluation benchmarks, which largely assume short search horizons and static answers.
Mind2Web 2 (arXiv:2506.21506) addresses this gap with two core contributions:
1. A benchmark of 130 realistic, high-quality, long-horizon tasks requiring real-time web browsing and extensive information synthesis, constructed with over 1,000 hours of human labor. 2. A novel Agent-as-a-Judge evaluation framework that constructs task-specific judge agents based on a tree-structured rubric design to automatically assess both answer correctness and source attribution.
Key Findings
- The authors conduct a comprehensive evaluation of ten frontier agentic search systems alongside human performance, with a detailed error analysis.
- The best-performing system, OpenAI Deep Research, already achieves 50–70% of human performance while spending half the time, highlighting strong potential of the paradigm.
- Mind2Web 2 provides a rigorous foundation for developing and benchmarking the next generation of agentic search systems.
- Evaluation shifts from static to dynamic answers: Time-varying, synthesis-heavy answers cannot be scored by exact-match or static reference comparisons; the tree-structured rubric enables automated, fine-grained judgment of correctness and attribution.
- Process-aware metrics: Beyond task success, source attribution quality becomes a first-class evaluation target—critical for trustworthy AI search.
- Realistic task design: The 130 tasks reflect genuine long-horizon information needs rather than short lookup queries.
- Agentic Information Retrieval (arXiv:2410.09713)
- Synergizing RAG and Reasoning: A Systematic Review (arXiv:2504.15909)
- AceSearcher: Bootstrapping Reasoning and Search for LLMs via Reinforcement Learning (arXiv:2509.24193)
- Boyu Gou, Zanming Huang, Yuting Ning, Yu Gu, Michael Lin, Weijian Qi, et al. (26 authors). *Mind2Web 2: Evaluating Agentic Search with Agent-as-a-Judge*, 2025. https://arxiv.org/abs/2506.21506
Why It Matters
Original Abstract
> Agentic search such as Deep Research systems—where agents autonomously browse the web, synthesize information, and return comprehensive citation-backed answers—represents a major shift in how users interact with web-scale information. While promising greater efficiency and cognitive offloading, the growing complexity and open-endedness of agentic search have outpaced existing evaluation benchmarks and methodologies, which largely assume short search horizons and static answers. In this paper, we introduce Mind2Web 2, a benchmark of 130 realistic, high-quality, and long-horizon tasks that require real-time web browsing and extensive information synthesis, constructed with over 1000 hours of human labor. To address the challenge of evaluating time-varying and complex answers, we propose a novel Agent-as-a-Judge framework. Our method constructs task-specific judge agents based on a tree-structured rubric design to automatically assess both answer correctness and source attribution. We conduct a comprehensive evaluation of ten frontier agentic search systems and human performance, along with a detailed error analysis to draw insights for future development. The best-performing system, OpenAI Deep Research, can already achieve 50-70% of human performance while spending half the time, highlighting its great potential. Altogether, Mind2Web 2 provides a rigorous foundation for developing and benchmarking the next generation of agentic search systems.