DeepDive: Teaching AI Deep Search via Knowledge Graphs and Multi-Turn RL

Forum topic · QianXun · 2025-11-25

Summary

DeepDive is a framework from Tsinghua University researchers that trains open-source LLMs to perform deep search—multi-step web research previously dominated by proprietary systems like OpenAI's DeepResearch. It combines two innovations: (1) automated data synthesis via random walks on knowledge graphs, followed by entity obfuscation and difficulty filtering with frontier LLMs, yielding 3,250 hard question-answer pairs; and (2) end-to-end multi-turn reinforcement learning using GRPO with a Jaccard-based redundancy penalty that discourages repeated queries. Applied to QwQ-32B, DeepDive raises BrowseComp accuracy from 1.3% to 15.3%, surpassing WebSailor-32B and DeepSeek-R1, while showing test-time scaling (success rising from 8% to 15% as tool calls grow from 8 to 128) and strong generalization on easier benchmarks like WebWalker (63.9%). Ablations confirm that data difficulty and format rewards are critical. The team openly notes a remaining gap to OpenAI's DeepResearch (51.5%) due to synthetic-data difficulty ceilings and over-search behaviors. Code, models, and data are open-sourced on GitHub.

Overview

DeepDive (Tsinghua University, 2025) is a framework that upgrades open-source LLMs from shallow retrieval into deep search agents capable of browsing dozens of web pages, filtering, and synthesizing information—closing part of the gap with OpenAI's DeepResearch and Google's Gemini Deep Research on the BrowseComp benchmark.

Key points

The problem: On BrowseComp-style deep search questions, strong open models fail badly—DeepSeek-R1 scores only 2–3%, versus >50% for OpenAI's DeepResearch. The gap reflects lack of search patience and strategy, plus a scarcity of hard training data.
Key 1 — Data synthesis: DeepDive performs random walks of 5–9 hops on knowledge graphs, then uses a frontier LLM (Gemini-2.5-Pro) to obscure entities into descriptive attributes, converting searchable facts into hard multi-hop puzzles. A difficulty filter keeps only questions that GPT-4o with basic search fails 4/4 times. This pipeline produced 3,250 high-quality QA pairs at a fraction of manual annotation cost.
Key 2 — Multi-turn RL: End-to-end RL over full trajectories (reasoning → search action → observation) using GRPO, with a Jaccard-similarity redundancy penalty that discourages repeated near-identical queries:

r'(\mathcal{T}) = r(\mathcal{T}) - \lambda \cdot S(\mathcal{T})

This cut tool calls by ~14% and improved late-stage accuracy by 20%.

Results

QwQ-32B base: 1.3% on BrowseComp → 9.5% after SFT → 15.3% after RL, beating WebSailor-32B (10.5%) and DeepSeek-R1 (2.0%) among open models.
Test-time scaling: success rises from 8% to 15% as max tool calls increase from 8 to 128.
Parallel sampling with least-tools selection: accuracy jumps from 12.0% to 24.8% across 8 runs—fewer searches correlate with model confidence.
Generalization: DeepDive-32B beats GPT-4o and Claude-3.7-Sonnet on easier benchmarks (e.g., 63.9% on WebWalker vs. DeepSeek-R1's 38.6%).

Ablations

Removing format rewards stalls learning (~8% plateau); with them, curves rise ~2 points higher throughout.
Training on HotpotQA yields weak RL results (9.2%) vs. synthetic deep-search data (12.0%): data difficulty determines capability ceiling.

Limitations

Still far behind OpenAI DeepResearch (51.5%); synthetic data cannot fully match the difficulty of expert-crafted benchmarks.
Over-search on easy questions suggests the need for difficulty-adaptive reward design.
n-gram contamination checks found >97% of training samples under 20% test overlap; none reached severe contamination.

Open source

Code, models, and data are released on GitHub. The project builds on GLM-4 and QwQ models, the Slime RL framework, and Serper/Jina APIs.

References

1. Lu, R., et al. (2025). *DeepDive: Advancing Deep Search Agents with Knowledge Graphs and Multi-Turn RL*. arXiv:2509.10446. 2. Wei, J., et al. (2025). *BrowseComp: A benchmark for deep search agents*. 3. Shao, Z., et al. (2024). *Deepseekmath: Pushing the limits of mathematical reasoning in open language models*. 4. Yao, S., et al. (2023). *ReAct: Synergizing reasoning and acting in language models*. 5. DeepSeek-AI, et al. (2025). *Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning*.