DeepResearcher: Training AI Research Agents via RL on the Real Internet

> Paper: Yuxiang Zheng et al., "DeepResearcher: Scaling Deep Research via Reinforcement Learning in Real-World Environments", arXiv:2504.03160, 2025

The Core Question

R1-Searcher learned to search in a simulated environment. But if you throw an AI directly onto the real internet — real Google Search, real web pages, real anti-scraping mechanisms, real loading delays — can it still learn?

DeepResearcher's answer: yes, but the entire training infrastructure has to be redesigned.

What the Paper Actually Does

The paper comes from a Huawei and Shanghai Jiao Tong University team. They did what R1-Searcher did not: end-to-end training of a Deep Research agent in a real web environment.

Core architecture:

• Environment: real Google Search API + web browser (Playwright)
• Models: 7B to 32B LLMs
• Algorithm: GRPO (Group Relative Policy Optimization)
• Rewards: outcome reward + process reward
• Scale: thousands of research trajectories across multiple domains

Key differences from R1-Searcher:

| Dimension | R1-Searcher | DeepResearcher | |------|------------|----------------| | Environment | Simulated retrieval (static dataset) | Real web (Google Search + browser) | | Information source | Pre-built document collection | Live internet content | | Training cost | Low (no API calls) | High (many search/browse calls) | | Realism | Low | High | | Scalability | High | Limited by API quotas and budget |

A Feynman-Style Analogy: Simulated vs. Real

Imagine training a pilot.

R1-Searcher's approach is like training in a flight simulator: preset airports, preset weather, preset failure modes. Cheap and repeatable, but the real world contains surprises the simulator never programmed — a bird strike, an unfamiliar controller, unexpected runway construction.

DeepResearcher's approach is like putting the trainee on a real aircraft over real routes. Expensive, risky, and every session burns real fuel — but the resulting pilot doesn't panic when reality deviates.

Both approaches have merit, but if you're deploying a real product, you must ask: how many hours of "real flight" does a simulator-trained pilot need? DeepResearcher skips that question entirely by training in the real environment. The cost:

• Slow training (every query waits on search engine responses)
• High cost (API call fees)
• Instability (web content changes; the same query returns different results today vs. tomorrow)
• Non-reproducibility (a training run can't be exactly replayed because the web keeps changing)

Key Findings

1. Real-environment training genuinely improves generalization. On held-out test sets (domains unseen during training), DeepResearcher significantly outperforms models trained only in simulated environments. The "noise" and diversity of the real web help the model learn more robust search strategies.

2. The model spontaneously learns "information verification." In simulated environments, models tend to trust the first batch of retrieved results. In the real environment, the model gradually learns to:

• Query the same question multiple ways
• Compare information across sources
• Detect contradictions and actively seek more evidence

Nobody explicitly taught this — it emerged naturally from RL, because on a noisy real web, "blindly trusting the first result" earns very low rewards.

3. Training cost is claimed acceptable — but not quantified. The paper claims that with careful batching and caching, real-environment training cost is "within acceptable range," yet provides no concrete numbers. This is an opacity in the paper.

The Real Insight

In my view, DeepResearcher's most valuable contribution is not the idea of "training in real environments" — that's obviously a good idea. It's proving that the credit-assignment problem for RL in real environments is solvable.

This is a hard engineering problem. In a real environment:

• A research trajectory may contain 20–50 steps
• Each step's quality depends on the final outcome
• Network latency and uncertainty make "which step contributed most" extremely ambiguous

DeepResearcher uses GRPO's group-relative baseline to effectively reduce the variance of credit assignment. This isn't a new algorithm — GRPO came from DeepSeek-R1 — but DeepResearcher demonstrates GRPO's effectiveness in complex real-world settings.

The deeper question: does this training approach scale? Current training covers a few domains (encyclopedic, news, tech). Covering all human knowledge would require massive real-world interaction, and Google Search API rate limits and quotas create an infrastructure bottleneck. The paper doesn't discuss this, but for industry it's the most critical question.

A Critical Perspective

A Feynman-style challenge: "Is this system really 'researching,' or is it strategically querying a search engine and assembling answers?"

The distinction matters. Genuine research involves:

• Proposing original hypotheses
• Designing verification methods
• Deriving new insights from results
• Identifying gaps in existing knowledge

What DeepResearcher mainly does: given a question, find relevant information, then synthesize an answer. That is closer to "advanced information retrieval" than "research." Calling the system a "Deep Researcher" may itself be a kind of cargo cult — a impressive-sounding label for what is functionally an advanced search agent.

This raises a more fundamental question: has the term "Deep Research" been overused? If "look things up and write a summary" counts as Deep Research, what do human graduate students do?

Conclusion

DeepResearcher is a key step from "toy demo" to "real product" for Deep Research. It demonstrates: 1. Training in a real web environment is feasible 2. RL can make models spontaneously learn information-verification strategies 3. Real-environment noise aids generalization

But it also exposes the field's core challenges:

• How to control training cost?
• How to break through the infrastructure bottleneck?
• Does the definition of "research" need recalibration?

For engineers: if you're building a real Deep Research product, DeepResearcher's training paradigm is worth studying — but be prepared for engineering complexity far beyond simulated environments.

For researchers: the paper poses a deeper question — when we say "AI is doing research," what do we mean? Is retrieval + synthesis research? Without "original hypothesis generation" and "experimental verification," are these systems essentially just more efficient information-organizing tools?

> "The first principle is that you must not fool yourself." By calling an "advanced search agent" a "Deep Researcher," are we fooling ourselves?

---

*Hashtags from the original post: #deep-research #paper-review #DeepResearcher #AgenticRAG #real-world-training #reinforcement-learning #Huawei #SJTU #Feynman-perspective*