This post introduces DeepWeb-Bench: A Deep Research Benchmark Demanding Massive Cross-Source Evidence and Long-Horizon Derivation, a paper by Sixiong Xie, Zhuofan Shi, Haiyang Shen, et al. from Peking University (arXiv 2605.15830, May 2026).
The author opens with an analogy: hiring a private detective who returns fifty polished reports in half an hour, yet produces a precise-looking but logically wrong figure—or outright fabricates one—when asked for the company's per-vehicle profit. This 'great at finding sources, terrible at drawing conclusions' gap, the post argues, is the core weakness of today's top AI models in deep research.
The Three Hurdles of Deep Research
Simple factual lookups are easy for current AI, but real deep research requires:
1. Massive evidence collection: extracting fragmentary information from dozens or hundreds of web pages. 2. Cross-source reconciliation: unifying conflicting data (e.g., financial reports in RMB vs. news in USD) and filtering noise. 3. Long-horizon multi-step derivation: the answer is buried in the data but requires complex arithmetic and nested logic to extract.
Key Findings: Retrieval Is Solved, Reasoning Is Not
The team evaluated 9 leading models, including the GPT-5 series and Claude 4, with a striking conclusion:
- Retrieval failures account for only 12–14% of total errors. AI agents are already excellent 'research assistants' at locating raw evidence.
- Derivation and calibration failures exceed 70%.
- Mid-tier models suffer from 'fake precision.' When exact data is missing, they confidently fabricate plausible numbers down to two decimal places—a hallmark of weaker models.
- Top models fail through 'incomplete derivation.' They gather all the raw materials but stumble at the final step. For example, when computing a carmaker's per-vehicle net profit, they correctly retrieve total revenue, total profit, and sales volume, but then wrongly use total gross profit instead of automotive-business gross profit.
Two failure patterns emerge:
The Unopened 'Derivation Black Box'
The post notes a remaining open problem: how do models implicitly balance conflicting evidence? When one page says A and another says B, is the model's internal attention mechanism weighing source authority, information frequency, or something unexplained? The paper introduces the notion of 'calibration,' but the weighting logic during derivation still lacks transparent mathematical explanation.
Takeaway
The benchmark marks a shift in evaluation: no longer satisfied with testing AI's memory or search ability, the field now demands that AI evolve from a 'diligent porter' into a 'rigorous analyst.'
> Truth often lies not at the top of a search box, but emerges from deep reorganization of evidence.
Next time an AI hands you a 10,000-word research report, don't just marvel at the speed—interrogate the final step of its reasoning, because behind a confident answer may hide a logically broken 'hollow chain.'