EXSEARCH: Iterative Self-Incentivization Empowers LLMs as Agentic Searchers

Paper: Iterative Self-Incentivization Empowers Large Language Models as Agentic Searchers Authors: Zhengliang Shi, Lingyong Yan, Dawei Yin, Suzan Verberne, Maarten de Rijke, Zhaochun Ren Published: 2025-05-26 Source: https://arxiv.org/abs/2505.20128

TL;DR

The paper proposes EXSEARCH, an agentic search framework where an LLM learns to retrieve useful information as its reasoning unfolds, through a self-incentivized iterative training process built on a Generalized Expectation-Maximization algorithm.

Background and Motivation

LLMs have been widely integrated into information retrieval, but enabling them to seek accurate knowledge in complex tasks remains challenging due to multi-hop query complexity and irrelevant retrieved content. Traditional pipelines treat retrieval, ranking, and generation separately, which struggles to meet modern demands for natural-language interaction, multi-hop reasoning, and real-time knowledge. EXSEARCH addresses the gap between one-shot retrieval and reasoning-aware, agentic search.

Method: EXSEARCH

At each step, the LLM: 1. Thinking — decides what to retrieve next; 2. Search — triggers an external retriever; 3. Recording — extracts fine-grained evidence to support next-step reasoning.

Training adopts a Generalized Expectation-Maximization algorithm:

• E-step: the LLM generates multiple search trajectories and assigns an importance weight to each;
• M-step: the LLM is trained on these trajectories with a re-weighted loss function.

This creates a self-incentivized loop: the LLM iteratively learns from its own generated data, progressively improving its search capability. The authors provide theoretical convergence guarantees for this training process.

Results

• Evaluated on four knowledge-intensive benchmarks;
• Substantially outperforms baselines, e.g., +7.8% improvement on exact match score;
• Introduces EXSEARCH-Zoo, an extension to broader scenarios to facilitate future work.

*Note: consult the original PDF for full tables and statistical details before citing quantitative results.*

Original Abstract

> Large language models (LLMs) have been widely integrated into information retrieval to advance traditional techniques. However, effectively enabling LLMs to seek accurate knowledge in complex tasks remains a challenge due to the complexity of multi-hop queries as well as the irrelevant retrieved content. To address these limitations, we propose EXSEARCH, an agentic search framework, where the LLM learns to retrieve useful information as the reasoning unfolds through a self-incentivized process. At each step, the LLM decides what to retrieve (thinking), triggers an external retriever (search), and extracts fine-grained evidence (recording) to support next-step reasoning. To enable LLM with this capability, EXSEARCH adopts a Generalized Expectation-Maximization algorithm. In the E-step, the LLM generates multiple search trajectories and assigns an importance weight to each; the M-step trains the LLM on them with a re-weighted loss function. This creates a self-incentivized loop, where the LLM iteratively learns from its own generated data, progressively improving itself for search. We further theoretically analyze this training process, establishing convergence guarantees. Extensive experiments on four knowledge-intensive benchmarks show that EXSEARCH substantially outperforms baselines, e.g., +7.8% improvement on exact match score. Motivated by these promising results, we introduce EXSEARCH-Zoo, an extension that extends our method to broader scenarios, to facilitate future work.

Key Takeaways for Search / Rec / Personalization

1. Architecture: cascaded retrieval + rerank + generation remains mainstream, but the agentic paradigm makes "when and what to retrieve" itself learnable; 2. Data: high-quality instruction data and session logs are critical; synthetic data must guard against knowledge leakage and distribution shift; 3. Evaluation: the gap between offline metrics and online satisfaction is widening; LLM-as-judge needs cross-validation with human evaluation; 4. Deployment: latency, cost, explainability, and safety are hard constraints — do not optimize only academic benchmarks.

Limitations and Future Work

Possible limitations include experiment scale constrained by GPU budget, benchmark/real-user distribution mismatch, English-centric data with unknown cross-lingual generalization, and security risks of open-web agentic retrieval. Future directions include more efficient test-time compute allocation, deeper integration with knowledge graphs and structured databases, and causal/fairness constraints for recommendation systems.

Related Reading

• Synergizing RAG and Reasoning: A Systematic Review (arXiv 2504.15909)
• Agentic Information Retrieval (arXiv 2410.09713)
• AceSearcher: Bootstrapping Reasoning and Search for LLMs via RL (arXiv 2509.24193)