DeepResearcher: Training AI Research Agents via Reinforcement Learning on the Real Web

> Paper: Yuxiang Zheng et al., "DeepResearcher: Scaling Deep Research via Reinforcement Learning in Real-World Environments", arXiv:2504.03160, 2025

The Core Question

R1-Searcher learned to search in simulated environments. But can an AI still learn when thrown directly onto the real internet—real Google Search, real web pages, real anti-scraping mechanisms, real loading latency?

DeepResearcher's answer: yes, but the entire training infrastructure has to be redesigned.

What the Paper Actually Does

The paper comes from a Huawei and Shanghai Jiao Tong University team. They did what R1-Searcher did not: end-to-end training of a Deep Research agent in a real web environment.

Core architecture:

• Environment: real Google Search API + web browser (Playwright)
• Models trained: 7B to 32B LLMs
• Training algorithm: GRPO (Group Relative Policy Optimization)
• Reward: outcome correctness + process reward
• Scale: thousands of research trajectories across multiple domains

Key differences from R1-Searcher:

| Dimension | R1-Searcher | DeepResearcher | |------|------------|----------------| | Environment | Simulated retrieval (static dataset) | Real web (Google Search + browser) | | Information source | Pre-built document collection | Live internet content | | Training cost | Low (no API calls) | High (many search/browse calls) | | Realism | Low | High | | Scalability | High | Limited by API quotas and budget |

A Feynman-Style Analogy: Simulated vs. Real

Suppose you want to train a pilot.

R1-Searcher's approach is like training in a flight simulator: preset airports, preset weather, preset failure modes. The pilot can practice takeoffs and landings hundreds of times at near-zero cost. But the real world contains surprises not programmed into the simulator—a bird strike, an ATC controller with a thick accent, unexpected runway construction.

DeepResearcher's approach is like putting the trainee straight into a real aircraft on real routes. Expensive, risky, burns real fuel every time. But the pilot trained this way doesn't panic when facing real surprises.

Both approaches have merit. But if you're deploying a real product (not a research prototype), you must answer: how many hours of real flying does a simulator-trained pilot need before going live?

DeepResearcher skips that question entirely—it trains directly in the real environment. The cost:

• Slow training (every query waits on a search engine response)
• High cost (API call fees)
• Instability (web content changes; the same query returns different results today vs. tomorrow)
• Non-reproducibility (a training run can't be exactly replicated because the web environment keeps changing)

Key Findings

1. Real-environment training genuinely improves generalization. On held-out test sets (domains unseen during training), DeepResearcher significantly outperforms models trained only in simulated environments. The noise and diversity of the real web apparently help the model learn more robust search strategies.

2. The model learns "information verification" behavior. In simulated environments, models tend to trust the first batch of retrieved results. But in the real environment, the model gradually learns to:

• Query the same question multiple ways
• Compare information across different sources
• Detect contradictions and actively seek more evidence

Nobody explicitly taught this—RL naturally evolved it in the real environment. Because the web is so noisy and inaccurate, the strategy of "blindly trusting the first results" earns low reward.

3. Training cost is claimed to be acceptable. The paper says that with careful batching and caching, real-environment training cost is "within an acceptable range." But no concrete numbers are given—an opacity in the paper.

The Real Insight

In my view, DeepResearcher's most valuable contribution is not the idea of "training in a real environment" itself—that's obviously a good idea. It is this: it proves the RL credit assignment problem in real environments is solvable.

This is a massive engineering problem. In a real environment:

• A research trajectory may contain 20–50 steps
• Each step's quality depends on the final outcome
• But network latency and uncertainty make it extremely ambiguous which step contributed most to the result

DeepResearcher uses GRPO's group-relative baseline to effectively reduce the variance of credit assignment. This isn't a novel algorithmic contribution—GRPO was proposed by DeepSeek-R1—but DeepResearcher demonstrates GRPO's effectiveness in complex real-world settings.

The deeper question: can this training method scale?

Current training covers a few domains (encyclopedic knowledge, news, tech). What about covering all human knowledge? Each domain requires massive real-environment interaction, and the Google Search API has rate limits and quotas—a infrastructure bottleneck. The paper doesn't discuss this, but for industry it's the most critical question.

A Critical Perspective

A Feynman-style challenge: "Is this system actually 'researching'? Or is it strategically querying a search engine and stitching together answers?"

The distinction matters. Real research includes:

• Proposing original hypotheses
• Designing validation methods
• Deriving new insights from results
• Identifying gaps in existing knowledge

What DeepResearcher mainly does: given a question, find relevant information, then synthesize an answer. That's closer to "advanced information retrieval" than "research."

Calling the system a "Deep Researcher" may itself be cargo-cult naming—an impressive label for capabilities closer to an "advanced search agent."

This raises a more fundamental question: has the term "Deep Research" been overused? If "look things up and write a summary" counts as Deep Research, what do human graduate students do?

Conclusion

DeepResearcher is a key step for Deep Research moving from "toy demo" toward "real product." It demonstrates: 1. Training in a real web environment is feasible 2. RL lets models spontaneously learn information-verification strategies 3. Real-environment noise aids generalization

But it also exposes the field's core challenges:

• How to control training costs?
• How to break through the infrastructure bottleneck?
• Does the definition of "research" need recalibration?

For engineers: if you're building a real Deep Research product, DeepResearcher's training paradigm is worth studying—but be prepared, real-environment training carries far more engineering complexity than simulation.

For researchers: this paper poses a deeper question—when we say "AI is doing research," what do we mean? Is retrieval + synthesis = research? Without "original hypothesis generation" and "experimental validation," are these systems essentially just more efficient information-organizing tools?

> "The first principle is that you must not fool yourself." By calling an "advanced search agent" a "Deep Researcher," are we fooling ourselves?