EXSEARCH: Iterative Self-Incentivization Empowers LLMs as Agentic Searchers

This is a forum digest of the paper "Iterative Self-Incentivization Empowers Large Language Models as Agentic Searchers" (arXiv, May 2025), listed under the Agentic Search section.

• Paper: https://arxiv.org/abs/2505.20128
• Authors: Zhengliang Shi, Lingyong Yan, Dawei Yin, Suzan Verberne, Maarten de Rijke, Zhaochun Ren
• Date: 2025-05-26

Key points

• Problem: LLMs integrated into information retrieval still struggle with complex multi-hop queries and irrelevant retrieved content; effectively seeking accurate knowledge during reasoning remains an open challenge.
• Approach — EXSEARCH: an agentic search framework where, at each step, the LLM decides what to retrieve (thinking), triggers an external retriever (search), and extracts fine-grained evidence (recording) to support next-step reasoning.
• Training — Generalized EM: in the E-step, the LLM generates multiple search trajectories and assigns importance weights to each; in the M-step, it trains on them with a re-weighted loss function. This self-incentivized loop lets the LLM iteratively learn from its own generated data.
• Theory: the paper provides a theoretical analysis of the training process with convergence guarantees.
• Results: on four knowledge-intensive benchmarks, EXSEARCH substantially outperforms baselines, e.g., +7.8% improvement on exact match score.
• EXSEARCH-Zoo: an extension applying the method to broader scenarios to facilitate future work.

Original abstract (quoted)

> Large language models (LLMs) have been widely integrated into information retrieval to advance traditional techniques. However, effectively enabling LLMs to seek accurate knowledge in complex tasks remains a challenge due to the complexity of multi-hop queries as well as the irrelevant retrieved content. To address these limitations, we propose EXSEARCH, an agentic search framework, where the LLM learns to retrieve useful information as the reasoning unfolds through a self-incentivized process. At each step, the LLM decides what to retrieve (thinking), triggers an external retriever (search), and extracts fine-grained evidence (recording) to support next-step reasoning. To enable LLM with this capability, EXSEARCH adopts a Generalized Expectation-Maximization algorithm. In the E-step, the LLM generates multiple search trajectories and assigns an importance weight to each; the M-step trains the LLM on them with a re-weighted loss function. This creates a self-incentivized loop, where the LLM iteratively learns from its own generated data, progressively improving itself for search. We further theoretically analyze this training process, establishing convergence guarantees. Extensive experiments on four knowledge-intensive benchmarks show that EXSEARCH substantially outperforms baselines, e.g., +7.8% improvement on exact match score. Motivated by these promising results, we introduce EXSEARCH-Zoo, an extension that extends our method to broader scenarios, to facilitate future work.

Context and significance

EXSEARCH sits at the intersection of agentic search and large-scale search systems. Rather than a fixed retrieve-then-generate pipeline, it treats the retrieval policy itself — whether, when, and what to search — as a learnable component, aligned with the broader shift toward iterative, verifiable, and planned knowledge access in RAG systems. Related entries in this collection include work on agentic information retrieval and reinforcement-learning-based search bootstrapping (e.g., AceSearcher).

Caveats

Quantitative details beyond the abstract (per-benchmark numbers, ablations, compute cost) should be verified against the original PDF. As with most arXiv work, cross-lingual generalization, online latency/cost constraints, and safety of open-web retrieval remain open questions.

References

• Iterative Self-Incentivization Empowers Large Language Models as Agentic Searchers. arXiv:2505.20128, https://arxiv.org/abs/2505.20128