AceSearcher: Bootstrapping Reasoning and Search for LLMs via Reinforced Self-Play
Paper: AceSearcher: Bootstrapping Reasoning and Search for LLMs via Reinforced Self-Play Authors: Ran Xu, Yuchen Zhuang, Zihan Dong, Jonathan Wang, Yue Yu, Joyce C. Ho, et al. (10 authors) Published: 2025-09-29 Source: https://www.arxiv.org/abs/2509.24193 Category: Agentic Search
TL;DR
AceSearcher trains a single LLM to play two cooperating roles — a query decomposer and an answer solver — via reinforced self-play, improving multi-hop retrieval and complex reasoning without intermediate annotations.
Background and Motivation
Search-augmented LLMs often struggle with complex reasoning tasks due to ineffective multi-hop retrieval and limited reasoning ability. Traditional pipelines separate retrieval, ranking, and generation, which poorly fits the modern demands of natural-language interaction, multi-hop reasoning, and up-to-date knowledge. AceSearcher addresses this gap by unifying decomposition and solving within one self-play-trained model.
Core Contributions
- A cooperative self-play framework where one LLM alternates between decomposing complex queries and integrating retrieved contexts for answer generation.
- A training recipe combining supervised fine-tuning on a mixture of search, reasoning, and decomposition tasks with reinforcement fine-tuning optimized only on final answer accuracy — no intermediate annotations required.
- Strong empirical results across 10 datasets and three reasoning-intensive tasks, plus discussion of interfaces with LLM tool calling, reinforcement learning, and multi-agent collaboration.
- Open problems identified: evaluation trustworthiness, latency and cost, hallucination and safety, and cross-lingual / multimodal extension.
- Code: https://github.com/ritaranx/AceSearcher
- Models: https://huggingface.co/AceSearcher
- Paper: https://www.arxiv.org/abs/2509.24193
Method Overview
The pipeline follows the standard arc of problem formalization → model design → training → inference:
1. Input & representation: encode queries, documents, and context as structured prompts. 2. Core modules: the same LLM acting as decomposer (sub-query planner) and solver (context integrator / generator). 3. Learning strategy: SFT on diverse search/reasoning/decomposition data, then reinforcement fine-tuning on final-answer accuracy. 4. Inference strategy: iterative retrieval with parallel sub-queries and budget control.
Results (from the abstract)
> Extensive experiments on three reasoning-intensive tasks across 10 datasets show that AceSearcher outperforms state-of-the-art baselines, achieving an average exact match improvement of 7.6%. Remarkably, on document-level finance reasoning tasks, AceSearcher-32B matches the performance of the DeepSeek-V3 model using less than 5% of its parameters. Even at smaller scales (1.5B and 8B), AceSearcher often surpasses existing search-augmented LLMs with up to 9x more parameters.
Note: for precise per-dataset numbers, consult the original PDF tables.
Insights for Search / RAG Practitioners
1. Architecture: cascade retrieval + reranking + generation remains mainstream, but agentic paradigms make "how many retrievals and what strategy" themselves learnable. 2. Data: high-quality instruction data and synthetic bootstrapping matter; guard against knowledge leakage and distribution shift. 3. Evaluation: offline metrics diverge from online satisfaction; cross-validate LLM-as-judge with human assessment. 4. Production: latency, cost, explainability, and safety are hard constraints — don't optimize only academic benchmarks.
Limitations and Future Work
Likely limitations include experiment scale constrained by compute budgets, benchmarks mismatching real user distributions, and unknown cross-lingual generalization. Future directions: more efficient test-time compute allocation, deeper integration with knowledge graphs and structured databases, and causal/fairness constraints for recommendation scenarios.