EXSEARCH: Iterative Self-Incentivization Empowers LLMs as Agentic Searchers

Overview

This post introduces the paper "Iterative Self-Incentivization Empowers Large Language Models as Agentic Searchers" by Zhengliang Shi, Lingyong Yan, Dawei Yin, Suzan Verberne, Maarten de Rijke, and Zhaochun Ren.

• arXiv: https://arxiv.org/abs/2505.20128
• Published: 2025-05-26
• Topic: Agentic Search

Problem

LLMs have been widely integrated into information retrieval, but enabling them to seek accurate knowledge in complex tasks remains difficult due to:

• The complexity of multi-hop queries
• Irrelevant retrieved content

Method: EXSEARCH

EXSEARCH is an agentic search framework in which the LLM learns to retrieve useful information as reasoning unfolds, via a self-incentivized process. At each step the LLM:

1. Thinking — decides what to retrieve 2. Search — triggers an external retriever 3. Recording — extracts fine-grained evidence to support next-step reasoning

Training uses a Generalized Expectation-Maximization (EM) algorithm:

• E-step: the LLM generates multiple search trajectories and assigns an importance weight to each
• M-step: the LLM is trained on these trajectories with a re-weighted loss function

This creates a self-incentivized loop where the model iteratively learns from its own generated data. The authors also provide theoretical analysis establishing convergence guarantees for the training process.

Results

• Evaluated on four knowledge-intensive benchmarks
• Substantially outperforms baselines, e.g., +7.8% improvement on exact match score

The authors additionally introduce EXSEARCH-Zoo, an extension of the method to broader scenarios to facilitate future work.

Original Abstract

> Large language models (LLMs) have been widely integrated into information retrieval to advance traditional techniques. However, effectively enabling LLMs to seek accurate knowledge in complex tasks remains a challenge due to the complexity of multi-hop queries as well as the irrelevant retrieved content. To address these limitations, we propose EXSEARCH, an agentic search framework, where the LLM learns to retrieve useful information as the reasoning unfolds through a self-incentivized process. At each step, the LLM decides what to retrieve (thinking), triggers an external retriever (search), and extracts fine-grained evidence (recording) to support next-step reasoning. To enable LLM with this capability, EXSEARCH adopts a Generalized Expectation-Maximization algorithm. In the E-step, the LLM generates multiple search trajectories and assigns an importance weight to each; the M-step trains the LLM on them with a re-weighted loss function. This creates a self-incentivized loop, where the LLM iteratively learns from its own generated data, progressively improving itself for search. We further theoretically analyze this training process, establishing convergence guarantees. Extensive experiments on four knowledge-intensive benchmarks show that EXSEARCH substantially outperforms baselines, e.g., +7.8% improvement on exact match score. Motivated by these promising results, we introduce EXSEARCH-Zoo, an extension that extends our method to broader scenarios, to facilitate future work.

Takeaways for Search / Rec / Personalization

• The agentic paradigm makes retrieval frequency and strategy themselves learnable objects, beyond the classic retrieve-rerank-generate cascade.
• Self-generated training data (with re-weighting) can progressively improve search behavior without new labeled trajectories.
• Convergence guarantees add theoretical grounding for EM-style self-improvement loops.
• Open questions remain around evaluation credibility, latency/cost, hallucination, and safety in open-web agentic settings.

Related Reading

• Synergizing RAG and Reasoning: A Systematic Review (arXiv:2504.15909)
• Agentic Information Retrieval (arXiv:2410.09713)
• AceSearcher: Bootstrapping Reasoning and Search for LLMs (arXiv:2509.24193)