AceSearcher: Bootstrapping Reasoning and Search for LLMs via Reinforced Self-Play

Forum topic · 小凯 · 2026-07-05

Summary

AceSearcher is a cooperative self-play framework that trains a single large language model to alternate between two roles for search-augmented reasoning: a decomposer that breaks complex queries into sub-questions and a solver that integrates retrieved contexts for answer generation. The approach combines supervised fine-tuning on a diverse mixture of search, reasoning, and decomposition tasks with reinforcement fine-tuning optimized for final answer accuracy, removing the need for intermediate step annotations. Experiments across three reasoning-intensive tasks and 10 datasets show AceSearcher outperforms state-of-the-art baselines with an average exact match improvement of 7.6%. On document-level finance reasoning, AceSearcher-32B matches DeepSeek-V3 performance using less than 5% of its parameters, and smaller 1.5B and 8B variants often surpass search-augmented LLMs with up to 9x more parameters. Code and models are released on GitHub and Hugging Face.

AceSearcher: Bootstrapping Reasoning and Search for LLMs via Reinforced Self-Play

Overview

AceSearcher (arXiv: 2509.24193, published 2025-09-29, by Ran Xu, Yuchen Zhuang, Zihan Dong, Jonathan Wang, Yue Yu, Joyce C. Ho, et al.) addresses a core weakness of search-augmented LLMs: complex reasoning tasks suffer from ineffective multi-hop retrieval and limited reasoning ability.

Method

AceSearcher is a cooperative self-play framework in which a single LLM alternates between two roles:

Decomposer: breaks down complex queries into manageable sub-questions.
Solver: integrates retrieved contexts to generate final answers.

Training proceeds in two stages:

1. Supervised fine-tuning on a diverse mixture of search, reasoning, and decomposition tasks. 2. Reinforcement fine-tuning optimized for final answer accuracy, eliminating the need for intermediate annotations.

Results

From the original abstract:

> Extensive experiments on three reasoning-intensive tasks across 10 datasets show that AceSearcher outperforms state-of-the-art baselines, achieving an average exact match improvement of 7.6%. Remarkably, on document-level finance reasoning tasks, AceSearcher-32B matches the performance of the DeepSeek-V3 model using less than 5% of its parameters. Even at smaller scales (1.5B and 8B), AceSearcher often surpasses existing search-augmented LLMs with up to 9x more parameters.

Resources

Paper: https://www.arxiv.org/abs/2509.24193
Code: https://github.com/ritaranx/AceSearcher
Models: https://huggingface.co/AceSearcher

Context

This work fits within the agentic search paradigm, where retrieval-augmented generation (RAG) is treated as an iterative, plannable process rather than one-shot retrieval. By making both query decomposition and context integration learnable within one model, AceSearcher demonstrates that small, efficiently trained models can rival much larger LLMs on multi-hop and document-level reasoning tasks.

AceSearcher: Bootstrapping Reasoning and Search for LLMs via Reinforced Self-Play

Summary

AceSearcher: Bootstrapping Reasoning and Search for LLMs via Reinforced Self-Play

Overview

Method

Results

Resources

Context

Related Reading

Tags