Mind2Web: Towards a Generalist Agent for the Web (NeurIPS 2023)

Overview

This forum entry indexes the NeurIPS 2023 (Datasets and Benchmarks track) paper "Mind2Web: Towards a Generalist Agent for the Web."

• Venue: NeurIPS 2023 Datasets and Benchmarks Track
• Category: Task Solving (agentic search / LLM agents)
• Link: https://proceedings.neurips.cc/paper_files/paper/2023/hash/5950bf290a1570ea401bf98882128160-Abstract-Datasets_and_Benchmarks.html

Key points

• Goal: Move beyond website-specific automation toward a *generalist* agent that executes web tasks from natural language instructions across many real-world sites and domains.
• Benchmark design: The dataset is built from live websites with human-verified task trajectories, spanning a broad range of domains and websites, enabling evaluation of generalization rather than memorization of a single site.
• Three generalization settings:
• *Cross-task*: new tasks on websites seen during training
• *Cross-website*: new websites within domains seen during training
• *Cross-domain*: entirely unseen domains
• Modeling contribution: The paper dissects LLMs' webpage understanding and next-action prediction (the 'Element Operation' formulation), and proposes a two-stage approach—candidate element ranking followed by action prediction—to make LLM-based action selection feasible given long webpage inputs. The resulting baseline is referred to as MindAct.
• Findings: LLMs show promise as web agents, but a significant gap remains between current models and human performance, especially under cross-website and cross-domain generalization, motivating this dataset as a stepping stone for generalist agents.

Context in the agentic search / IR landscape

The post situates Mind2Web at the intersection of agentic search and large-scale search/recommendation systems. Classic search pipelines treat retrieval, ranking, and generation as separate stages; LLM-era agents instead treat "when to retrieve, how many steps, and which tool to call" as part of the decision space. Web agents such as the one benchmarked by Mind2Web extend this by making the *browser itself* the action space.

Engineering considerations highlighted in the post include:

• Latency and cost budgets (cascaded retrieval, early stopping, caching)
• Online vs. offline evaluation gaps (task success rate vs. nDCG-style metrics)
• Safety risks of acting on the open web (prompt injection, unreliable sources)
• Data quality: instruction data, session logs, and synthetic data with leakage risks

Limitations noted by the community

• Benchmark-web gaps: recorded sites change over time, affecting reproducibility
• Evaluation initially relies on static snapshots / offline metrics rather than live interaction
• English-centric data; cross-lingual transfer unclear
• Safety concerns when deploying agents on the open web

Related entries

• Investigating Users' Search Behavior and Outcome with ChatGPT in Learning (DOI: 10.1145/3673791.3698406)

References

• Original paper: Mind2Web: Towards a Generalist Agent for the Web, NeurIPS 2023 — https://proceedings.neurips.cc/paper_files/paper/2023/hash/5950bf290a1570ea401bf98882128160-Abstract-Datasets_and_Benchmarks.html

*Note: Quantitative results should be verified against the original PDF; this entry is based on the abstract and public metadata of the paper.*