Mind2Web 2: Evaluating Agentic Search with Agent-as-a-Judge
Paper: arXiv:2506.21506 — Boyu Gou, Zanming Huang, Yuting Ning, Yu Gu, Michael Lin, Weijian Qi, et al. (26 authors), June 26, 2025.
Overview
Agentic search systems — such as Deep Research systems, where agents autonomously browse the web, synthesize information, and return comprehensive citation-backed answers — represent a major shift in how users interact with web-scale information. However, the growing complexity and open-endedness of agentic search have outpaced existing evaluation benchmarks, which largely assume short search horizons and static answers.
Key Contributions
- Mind2Web 2 benchmark: 130 realistic, high-quality, long-horizon tasks requiring real-time web browsing and extensive information synthesis, constructed with over 1,000 hours of human labor.
- Agent-as-a-Judge framework: A novel evaluation method that constructs task-specific judge agents based on a tree-structured rubric design, automatically assessing both answer correctness and source attribution for time-varying, complex answers.
- Comprehensive evaluation: Testing of ten frontier agentic search systems against human performance, plus detailed error analysis to guide future development.
- Task completion on open-ended, long-horizon web tasks
- Answer correctness via rubric-based judge agents
- Source attribution quality (citation-backed answers)
- Comparison against human baselines and cost/time budgets
Headline Result
The best-performing system, OpenAI Deep Research, achieves 50–70% of human performance while spending half the time, highlighting strong potential for agentic search systems.
Evaluation Design Notes
Unlike traditional IR benchmarks measured with static metrics (e.g., nDCG@10, MRR, Recall@k), Mind2Web 2 evaluates:
Implications
1. Architecture: Agentic paradigms make retrieval strategy itself a learnable, decision-making component rather than a fixed pipeline stage. 2. Evaluation: LLM-based judges need rubric structure and cross-validation with human assessment to be trustworthy for open-ended answers. 3. Deployment: Latency, cost, attribution quality, and safety remain hard constraints for production agentic search systems.
Limitations & Future Work
Potential limitations include benchmark–real-user distribution mismatch, English-centric data limiting cross-lingual generalization, and safety risks of agents operating on the open web. Future directions include more efficient test-time compute allocation and deeper integration with structured knowledge sources.
Original Abstract
> Agentic search such as Deep Research systems—where agents autonomously browse the web, synthesize information, and return comprehensive citation-backed answers—represents a major shift in how users interact with web-scale information. While promising greater efficiency and cognitive offloading, the growing complexity and open-endedness of agentic search have outpaced existing evaluation benchmarks and methodologies, which largely assume short search horizons and static answers. In this paper, we introduce Mind2Web 2, a benchmark of 130 realistic, high-quality, and long-horizon tasks that require real-time web browsing and extensive information synthesis, constructed with over 1000 hours of human labor. To address the challenge of evaluating time-varying and complex answers, we propose a novel Agent-as-a-Judge framework. Our method constructs task-specific judge agents based on a tree-structured rubric design to automatically assess both answer correctness and source attribution. We conduct a comprehensive evaluation of ten frontier agentic search systems and human performance, along with a detailed error analysis to draw insights for future development. The best-performing system, OpenAI Deep Research, can already achieve 50-70% of human performance while spending half the time, highlighting its great potential. Altogether, Mind2Web 2 provides a rigorous foundation for developing and benchmarking the next generation of agentic search systems.