DeepWeb-Bench: A Deep Research Benchmark Demanding Massive Cross-Source Evidence

Forum topic · 小凯 · 2026-05-22

Summary

DeepWeb-Bench (arXiv:2505.15982) is a new benchmark designed to be substantially harder than existing deep research evaluations for frontier language models. Deep research agents search the open web, collect evidence, and derive answers through extended reasoning, but current frontier products already saturate existing benchmarks. DeepWeb-Bench introduces difficulty through three data properties: massive evidence collection, cross-source reconciliation, and long-horizon multi-step derivation. These are organized into four capability families—Retrieval, Derivation, Reasoning, and Calibration—with results reported per family. Each reference answer includes source provenance with a four-level disclosure tier and usable cross-source checks, making scores auditable against underlying evidence. Evaluations across nine frontier models yield three findings: retrieval is not the bottleneck, accounting for only 12–14% of errors while derivation and calibration failures exceed 70%; strong and weak models fail qualitatively differently, with strong models showing incomplete derivations and weak models showing hallucinated precision; and models exhibit genuine domain specialization, with cross-model agreement of only ρ=0.61 and per-case divergence up to 18.8 percentage points. The public release includes data, scoring rubrics, and evaluation code.

论文概要

Field: Machine Learning Authors: Sixiong Xie, Zhuofan Shi, Haiyang Shen Published: 2025-05-20 arXiv: 2505.15982

Overview

Deep research, in which an agent searches the open web, collects evidence, and derives an answer through extended reasoning, is a prominent use case for frontier language models. However, frontier deep research products score so high on existing benchmarks that it is difficult to distinguish their capabilities from current evaluation data alone. DeepWeb-Bench addresses this by providing a substantially harder benchmark for the current frontier.

Why Is It Harder?

Difficulty comes from three properties of the data itself:

Massive evidence collection — each task requires gathering large amounts of evidence from the open web
Cross-source reconciliation — evidence must be coordinated and reconciled across multiple sources
Long-horizon multi-step derivation — answers require extended chains of reasoning

These three sources of difficulty are represented as four capability families: Retrieval, Derivation, Reasoning, and Calibration, with results reported sliced by family.

Auditable Scoring

Each reference answer is accompanied by source provenance with a four-level disclosure tier and usable cross-source checks, making benchmark scores easier to audit against the underlying evidence.

Key Findings

The benchmark was evaluated on nine frontier models, yielding three findings:

1. Retrieval is not the bottleneck. Retrieval failures account for only 12–14% of errors, while derivation and calibration failures account for over 70%. 2. Strong and weak models fail qualitatively differently. Errors by strong models are dominated by incomplete derivations, whereas weak models are dominated by hallucinated precision. 3. Models show genuine domain specialization. Cross-model agreement is only ρ=0.61, with per-case divergence reaching 18.8 percentage points.

Availability

The public benchmark release includes the data, scoring rubrics, and evaluation code.

Paper: arXiv:2505.15982