EXSEARCH: Iterative Self-Incentivization Empowers LLMs as Agentic Searchers
Paper: arXiv:2505.20128 (May 26, 2025)
Authors: Zhengliang Shi, Lingyong Yan, Dawei Yin, Suzan Verberne, Maarten de Rijke, Zhaochun Ren
Original Abstract
> Large language models (LLMs) have been widely integrated into information retrieval to advance traditional techniques. However, effectively enabling LLMs to seek accurate knowledge in complex tasks remains a challenge due to the complexity of multi-hop queries as well as the irrelevant retrieved content. To address these limitations, we propose EXSEARCH, an agentic search framework, where the LLM learns to retrieve useful information as the reasoning unfolds through a self-incentivized process. At each step, the LLM decides what to retrieve (thinking), triggers an external retriever (search), and extracts fine-grained evidence (recording) to support next-step reasoning. To enable LLM with this capability, EXSEARCH adopts a Generalized Expectation-Maximization algorithm. In the E-step, the LLM generates multiple search trajectories and assigns an importance weight to each; the M-step trains the LLM on them with a re-weighted loss function. This creates a self-incentivized loop, where the LLM iteratively learns from its own generated data, progressively improving itself for search. We further theoretically analyze this training process, establishing convergence guarantees. Extensive experiments on four knowledge-intensive benchmarks show that EXSEARCH substantially outperforms baselines, e.g., +7.8% improvement on exact match score. Motivated by these promising results, we introduce EXSEARCH-Zoo, an extension that extends our method to broader scenarios, to facilitate future work.
Key points
- Problem: LLMs struggle to seek accurate knowledge in complex multi-hop tasks, and retrieved content is often irrelevant to the reasoning needs at each step.
- Approach: EXSEARCH formulates agentic search as a three-action loop — *thinking* (deciding what to retrieve), *search* (triggering an external retriever), and *recording* (extracting fine-grained evidence).
- Training: A Generalized Expectation-Maximization (EM) algorithm drives a self-incentivized loop. The E-step samples multiple search trajectories with importance weights; the M-step optimizes a re-weighted loss over them.
- Theory: The training process comes with convergence guarantees.
- Results: Outperforms baselines on four knowledge-intensive benchmarks, with up to +7.8% exact match improvement.
- Extension: EXSEARCH-Zoo generalizes the method to broader scenarios.
- Scaling behavior with larger models and open-web retrievers
- Latency and cost of iterative search steps in production settings
- Robustness to retrieval poisoning and distribution shift
- Paper: <https://arxiv.org/abs/2505.20128>
- Related lines of work: RAG and reasoning synergy surveys, RL-based search agents (e.g., AceSearcher), agentic information retrieval.
Why it matters
Agentic search shifts retrieval from a one-shot, pipeline-fixed step to a learnable sequential decision process: *how many times to search, what to query, and what evidence to keep* become policy outputs of the model. Self-incentivized training reduces dependence on expensive annotated search trajectories, and the EM-style reweighting offers an alternative to standard RL post-training for tool-use agents.