ZeroSearch: Incentivize the Search Capability of LLMs without Searching
- Paper: https://arxiv.org/abs/2505.04588
- Authors: Hao Sun, Zile Qiao, Jiayan Guo, Xuanbo Fan, Yingyan Hou, Yong Jiang, et al. (10 authors)
- Published: 2025-05-07
- Effective information searching is essential for enhancing the reasoning and generation capabilities of large language models (LLMs). Recent research has used reinforcement learning (RL) to improve LLMs' search capabilities by interacting with live search engines, but this faces two major challenges: 1. Uncontrolled document quality: search engines return documents of unpredictable quality, introducing noise and instability into training. 2. Prohibitively high API costs: RL training requires frequent rollouts, potentially hundreds of thousands of search requests, severely constraining scalability.
- ZeroSearch is a novel RL framework that incentivizes LLMs' ability to use a real search engine using simulated searches during training.
- Training begins with lightweight supervised fine-tuning that transforms the LLM into a retrieval module capable of generating both useful and noisy documents in response to a query.
- During RL training, a curriculum-based rollout strategy incrementally degrades the quality of generated documents, progressively eliciting the model's reasoning ability via increasingly challenging retrieval scenarios.
- Results: a 3B LLM suffices as the retrieval module; a 7B retrieval module achieves performance comparable to a real search engine, and a 14B module even surpasses it. The method generalizes across base and instruction-tuned models of various sizes and is compatible with a wide range of RL algorithms.
- Enhancing Noise Robustness of Retrieval-Augmented Language Models
- FreshLLMs: Refreshing Large Language Models with Search Engine Augmentation
- Gorilla: Large Language Model Connected with Massive APIs
- Interleaving Retrieval with Chain-of-Thought Reasoning
- Long-form factuality in large language models
- ReSearch: Learning to Reason with Search for LLMs via RL
- ZeroSearch: Incentivize the Search Capability of LLMs without Searching. arXiv:2505.04588
Key points
Abstract (original)
> Effective information searching is essential for enhancing the reasoning and generation capabilities of large language models (LLMs). Recent research has explored using reinforcement learning (RL) to improve LLMs' search capabilities by interacting with live search engines in real-world environments. While these approaches show promising results, they face two major challenges: (1) Uncontrolled Document Quality: The quality of documents returned by search engines is often unpredictable, introducing noise and instability into the training process. (2) Prohibitively High API Costs: RL training requires frequent rollouts, potentially involving hundreds of thousands of search requests, which incur substantial API expenses and severely constrain scalability. To address these challenges, we introduce ZeroSearch, a novel RL framework that incentivizes the capabilities of LLMs to use a real search engine with simulated searches during training. Our approach begins with lightweight supervised fine-tuning to transform the LLM into a retrieval module capable of generating both useful and noisy documents in response to a query. During RL training, we employ a curriculum-based rollout strategy that incrementally degrades the quality of generated documents, progressively eliciting the model's reasoning ability by exposing it to increasingly challenging retrieval scenarios. Extensive experiments demonstrate that ZeroSearch effectively incentivizes the search capabilities of LLMs using a 3B LLM as the retrieval module. Remarkably, a 7B retrieval module achieves comparable performance to the real search engine, while a 14B retrieval module even surpasses it. Furthermore, it generalizes well across both base and instruction-tuned models of various parameter sizes and is compatible with a wide range of RL algorithms.