Dr. Zero: Self-Evolving Search Agents without Training Data

Paper: Dr. Zero: Self-Evolving Search Agents without Training Data — arXiv, 2026-01-11

Overview

As high-quality training data becomes increasingly difficult to obtain, data-free self-evolution has emerged as a promising paradigm: LLMs autonomously generate and solve complex problems to improve their own reasoning. However, multi-turn search agents struggle with data-free self-evolution due to limited question diversity and the substantial compute required for multi-step reasoning and tool use. Dr. Zero tackles both problems.

Key contributions

• Self-evolution feedback loop: A *proposer* generates diverse questions used to train a *solver* initialized from the same base model. As the solver improves, the proposer is incentivized to produce increasingly difficult yet still solvable tasks — an automated curriculum refining both agents.
• Hop-grouped relative policy optimization (HRPO): Clusters structurally similar questions to construct group-level baselines, minimizing the sampling overhead of evaluating each query's individual difficulty and solvability. This significantly reduces solver-training compute without compromising performance or stability.
• Data-free results: Extensive experiments show Dr. Zero matches or surpasses fully supervised search agents, proving that complex reasoning and search capabilities can emerge solely through self-evolution.

Original abstract (verbatim)

> As high-quality data becomes increasingly difficult to obtain, data-free self-evolution has emerged as a promising paradigm. This approach allows large language models (LLMs) to autonomously generate and solve complex problems, thereby improving their reasoning capabilities. However, multi-turn search agents struggle in data-free self-evolution due to the limited question diversity and the substantial compute required for multi-step reasoning and tool using. In this work, we introduce Dr. Zero, a framework enabling search agents to effectively self-evolve without any training data. In particular, we design a self-evolution feedback loop where a proposer generates diverse questions to train a solver initialized from the same base model. As the solver evolves, it incentivizes the proposer to produce increasingly difficult yet solvable tasks, thus establishing an automated curriculum to refine both agents. To enhance training efficiency, we also introduce hop-grouped relative policy optimization (HRPO). This method clusters structurally similar questions to construct group-level baselines, effectively minimizing the sampling overhead in evaluating each query's individual difficulty and solvability. Consequently, HRPO significantly reduces the compute requirements for solver training without compromising performance or stability. Extensive experiment results demonstrate that the data-free Dr. Zero matches or surpasses fully supervised search agents, proving that complex reasoning and search capabilities can emerge solely through self-evolution.

Context and significance

Dr. Zero sits at the intersection of agentic search and LLM post-training. Traditional search pipelines separate retrieval, ranking, and generation, while agentic approaches treat retrieval frequency and strategy themselves as learnable behaviors. Dr. Zero removes the dependency on curated instruction data entirely, making self-improving search agents more practical.

Takeaways

1. Architecture: Retrieval strategy is becoming a learnable object; proposer–solver co-evolution provides a scalable data-generation mechanism. 2. Compute efficiency: Grouping structurally similar tasks (HRPO) is a general trick for cutting RL training costs in multi-step agent settings. 3. Evaluation caution: As with all agent papers, verify quantitative results against the PDF before citing; offline benchmarks may not fully reflect real-world query distributions, latency, or cost constraints.

Related work

• AceSearcher: Bootstrapping Reasoning and Search for LLMs via Reinforcement Learning (arXiv:2509.24193)
• Agentic Information Retrieval (arXiv:2410.09713)
• Synergizing RAG and Reasoning: A Systematic Review (arXiv:2504.15909)
• Enterprise Knowledge Retrieval frameworks (arXiv:2512.05411)

Glossary

| Term | Meaning | |------|---------| | RAG | Retrieval-Augmented Generation | | HRPO | Hop-grouped relative policy optimization (introduced in this paper) | | Agentic Search | Modeling search as sequential decisions and tool calls by an agent | | nDCG | Normalized discounted cumulative gain, a ranking quality metric |

Limitations and open questions

Potential limitations include experiment scale constrained by GPU budgets, benchmarks that may not match real user distributions, and unknown cross-language generalization. Future directions include more efficient test-time compute allocation, deeper integration with knowledge graphs, and safety measures for agents operating over the open web.