Beyond Ten Turns: Unlocking Long-Horizon Agentic Search with Large-Scale Asynchronous RL
- Paper: Beyond Ten Turns: Unlocking Long-Horizon Agentic Search with Large-Scale Asynchronous RL
- arXiv: https://arxiv.org/abs/2508.07976
- Code & models: https://github.com/inclusionAI/ASearcher
- Posted: 2025-08-11
- Authors: Jiaxuan Gao, Wei Fu, Minyang Xie, Shusheng Xu, Chuyi He, Zhiyu Mei, et al. (8 authors)
- RL-trained prompt-based QwQ-32B agent: +78.0% Avg@4 on xBench and +34.3% on GAIA.
- ASearcher-Web-QwQ (simple agent design, no external LLMs): Avg@4 of 51.1 on xBench and 58.7 on GAIA, surpassing existing open-source 32B agents.
- With an external summary tool, the agent approaches commercial system performance via zero-shot transfer and test-time search.
- A Systematic Framework for Enterprise Knowledge Retrieval
- Synergizing RAG and Reasoning: A Systematic Review
- AceSearcher: Bootstrapping Reasoning and Search for LLMs via RL
- Agentic Information Retrieval (Oct 2024)
Abstract (original)
Recent advancements in LLM-based agents have demonstrated remarkable capabilities in handling complex, knowledge-intensive tasks by integrating external tools. Among diverse choices of tools, search tools play a pivotal role in accessing vast external knowledge. However, open-source agents still fall short of achieving expert-level Search Intelligence, the ability to resolve ambiguous queries, generate precise searches, analyze results, and conduct thorough exploration. Existing approaches fall short in scalability, efficiency, and data quality. For example, small turn limits in existing online RL methods, e.g. <=10, restrict complex strategy learning. This paper introduces ASearcher, an open-source project for large-scale RL training of search agents. Our key contributions include: (1) Scalable fully asynchronous RL training that enables long-horizon search while maintaining high training efficiency. (2) A prompt-based LLM agent that autonomously synthesizes high-quality and challenging QAs, creating a large-scale QA dataset. Through RL training, our prompt-based QwQ-32B agent achieves substantial improvements, with 78.0% and 34.3% Avg@4 gains on xBench and GAIA, respectively. Notably, our agent exhibits extreme long-horizon search, with tool calls exceeding 100 turns and output tokens exceeding 400k during training time. With a simple agent design and no external LLMs, ASearcher-Web-QwQ achieves Avg@4 scores of 51.1 on xBench and 58.7 on GAIA, surpassing existing open-source 32B agents. Finally, we also show that ASearcher-Web-QwQ could achieve performance of commercial systems using external summary tool in a zero-shot transfer manner and test-time search. We open-source our models, training data, and codes in https://github.com/inclusionAI/ASearcher.
Key Contributions
1. Scalable fully asynchronous RL training — removes the small-turn bottleneck of prior online RL methods (typically <=10 turns), enabling long-horizon search agent training while keeping efficiency high. 2. Autonomous QA data synthesis — a prompt-based LLM agent generates high-quality and challenging QA pairs at scale, addressing the data-quality bottleneck in agent RL. 3. Extreme long-horizon behavior — during training the agent performs tool calls exceeding 100 turns and generates over 400k output tokens.
Results
Why It Matters
Turn limits of around 10 turns have been a structural ceiling for RL-trained search agents: complex multi-hop research requires far longer exploration chains. ASearcher shows that fully asynchronous large-scale RL, combined with agent-synthesized training data, can push agents beyond this ceiling — and the full stack (models, training data, code) is open-sourced for reproduction and further research.