Mind2Web 2: Evaluating Agentic Search with Agent-as-a-Judge
- Paper: https://arxiv.org/abs/2506.21506
- Authors: Boyu Gou, Zanming Huang, Yuting Ning, Yu Gu, Michael Lin, Weijian Qi, et al. (26 authors)
- Published: 2025-06-26
- Benchmark: 130 realistic, high-quality, long-horizon tasks requiring real-time web browsing and extensive information synthesis; built with over 1,000 hours of human labor.
- Agent-as-a-Judge: A novel evaluation framework that constructs task-specific judge agents using a tree-structured rubric, automatically assessing both answer correctness and source attribution — addressing the difficulty of judging time-varying, complex answers.
- Evaluation: Ten frontier agentic search systems benchmarked against human performance, with detailed error analysis.
- Headline result: OpenAI Deep Research, the best-performing system, achieves 50–70% of human performance while spending half the time.
- Architecture: The agentic paradigm makes retrieval count and strategy themselves learnable, beyond classic retrieve–rerank–generate pipelines.
- Evaluation: Offline metrics diverge from real user satisfaction; LLM-based judges should be cross-validated with human assessment.
- Production: Latency, cost, interpretability, and safety (e.g., open-web poisoning/bias) remain hard constraints — not just benchmark scores.
- Open problems: evaluation trustworthiness, latency/cost budgets, hallucination, and cross-lingual/multimodal extension.
- A Systematic Framework for Enterprise Knowledge Retrieval
- Retrieval Augmented Generation and Understanding in Vision: A Survey
- Synergizing RAG and Reasoning: A Systematic Review
- AceSearcher: Bootstrapping Reasoning and Search for LLMs via RL
- Agentic Information Retrieval
Abstract (original)
> Agentic search such as Deep Research systems—where agents autonomously browse the web, synthesize information, and return comprehensive citation-backed answers—represents a major shift in how users interact with web-scale information. While promising greater efficiency and cognitive offloading, the growing complexity and open-endedness of agentic search have outpaced existing evaluation benchmarks and methodologies, which largely assume short search horizons and static answers. In this paper, we introduce Mind2Web 2, a benchmark of 130 realistic, high-quality, and long-horizon tasks that require real-time web browsing and extensive information synthesis, constructed with over 1000 hours of human labor. To address the challenge of evaluating time-varying and complex answers, we propose a novel Agent-as-a-Judge framework. Our method constructs task-specific judge agents based on a tree-structured rubric design to automatically assess both answer correctness and source attribution. We conduct a comprehensive evaluation of ten frontier agentic search systems and human performance, along with a detailed error analysis to draw insights for future development. The best-performing system, OpenAI Deep Research, can already achieve 50-70% of human performance while spending half the time, highlighting its great potential. Altogether, Mind2Web 2 provides a rigorous foundation for developing and benchmarking the next generation of agentic search systems.
Key Points
Context
Traditional search evaluation assumes short search horizons and static answers (e.g., nDCG on fixed corpora). Agentic search systems — which iteratively browse, plan, and synthesize — require process-aware metrics such as task success, citation accuracy, and multi-hop reasoning completeness. Mind2Web 2 addresses this gap.