Mind2Web 2: Evaluating Agentic Search with Agent-as-a-Judge
Overview
- Paper: Mind2Web 2: Evaluating Agentic Search with Agent-as-a-Judge
- Authors: Boyu Gou, Zanming Huang, Yuting Ning, Yu Gu, Michael Lin, Weijian Qi, et al. (26 authors total)
- Published: 2025-06-26
- Link: https://arxiv.org/abs/2506.21506
- Type: Academic paper (Agentic Search / Search Engine Evaluation)
- Benchmark: Mind2Web 2 contains 130 realistic, high-quality, long-horizon tasks that require real-time web browsing and extensive information synthesis, constructed with over 1,000 hours of human labor.
- Motivation: Agentic search systems (e.g., Deep Research) autonomously browse the web, synthesize information, and return citation-backed answers. Existing benchmarks assume short search horizons and static answers, and have been outpaced by the growing complexity and open-endedness of these systems.
- Evaluation method: A novel Agent-as-a-Judge framework builds task-specific judge agents based on a tree-structured rubric design to automatically assess both answer correctness and source attribution.
- Results: Comprehensive evaluation of ten frontier agentic search systems versus human performance, plus detailed error analysis. The best system, OpenAI Deep Research, achieves 50-70% of human performance while spending half the time.
- Original paper: Mind2Web 2: Evaluating Agentic Search with Agent-as-a-Judge (arXiv:2506.21506)
Key Points
Abstract (original)
> Agentic search such as Deep Research systems—where agents autonomously browse the web, synthesize information, and return comprehensive citation-backed answers—represents a major shift in how users interact with web-scale information. While promising greater efficiency and cognitive offloading, the growing complexity and open-endedness of agentic search have outpaced existing evaluation benchmarks and methodologies, which largely assume short search horizons and static answers. In this paper, we introduce Mind2Web 2, a benchmark of 130 realistic, high-quality, and long-horizon tasks that require real-time web browsing and extensive information synthesis, constructed with over 1000 hours of human labor. To address the challenge of evaluating time-varying and complex answers, we propose a novel Agent-as-a-Judge framework. Our method constructs task-specific judge agents based on a tree-structured rubric design to automatically assess both answer correctness and source attribution. We conduct a comprehensive evaluation of ten frontier agentic search systems and human performance, along with a detailed error analysis to draw insights for future development. The best-performing system, OpenAI Deep Research, can already achieve 50-70% of human performance while spending half the time, highlighting its great potential. Altogether, Mind2Web 2 provides a rigorous foundation for developing and benchmarking the next generation of agentic search systems.
Editorial Context and Takeaways
The post frames the paper within the broader agentic search landscape and offers these insights:
1. Architecture: Cascaded retrieval + reranking + generation remains mainstream, but the agentic paradigm makes retrieval count and strategy themselves learnable. 2. Evaluation: The gap between offline metrics and online satisfaction is widening; LLM-as-judge approaches should be cross-validated against human evaluation — Mind2Web 2's rubric-based judge agents are a step in this direction. 3. Deployment constraints: Latency, cost, explainability, and safety policies are hard constraints for production systems; academic benchmarks alone are insufficient. 4. Open problems: Evaluation trustworthiness, latency/cost trade-offs, hallucination and safety, and cross-lingual/multimodal generalization.
> Note: The source post includes some generic commentary and hypothetical experimental details (e.g., dataset/metric lists) that are not confirmed by the paper's abstract. Quantitative conclusions should be verified against the original PDF.