Towards Agentic Self-Learning LLMs in Search Environment

This is a forum review of the paper "Towards Agentic Self-Learning LLMs in Search Environment" (arXiv:2510.14253), published October 2025, categorized under Agentic Search.

One-line summary

The paper studies whether self-learning can scale LLM-based search agents without human-curated datasets or predefined rule-based rewards, and proposes Agentic Self-Learning (ASL), a closed-loop multi-role RL framework.

Background and Motivation

In large-scale search, recommendation, and personalization systems, agentic search has long faced challenges around efficiency, scalability, and user intent understanding. Traditional pipeline approaches treat retrieval, ranking, and generation as separate stages, which struggles to meet modern demands for natural-language interaction, multi-hop reasoning, and up-to-date knowledge. The paper addresses the core question of whether self-learning can scale LLM agents without relying on human-curated datasets or predefined rule-based rewards.

Core Contributions

• Identifies two key determinants of scalable agent training: the source of reward signals and the scale of agent task data.
• Shows that Generative Reward Model (GRM) rewards outperform rigid rule-based signals for open-domain learning, and that co-evolving the GRM with the policy further boosts performance.
• Demonstrates that increasing the volume of agent task data — even when synthetically generated — substantially enhances agentic capabilities.
• Proposes Agentic Self-Learning (ASL), a fully closed-loop, multi-role reinforcement learning framework unifying task generation, policy execution, and evaluation within a shared tool environment and LLM backbone.
• Identifies GRM verification capacity as the main bottleneck: a frozen GRM induces reward hacking, while continual GRM training mitigates this and a small late-stage injection of real verification data raises the performance ceiling.

Method / Architecture

ASL coordinates three roles in a virtuous cycle of harder task setting, sharper verification, and stronger solving:

1. Prompt Generator — creates agent tasks, including synthetic ones; 2. Policy Model — executes search-agent tasks; 3. Generative Reward Model — evaluates policy outputs and provides training rewards.

All roles share a common tool environment and LLM backbone, forming a closed feedback loop.

Experimental Findings

• ASL delivers steady, round-over-round gains, while strong RLVR baselines (e.g., Search-R1) plateau or degrade.
• ASL continues improving under zero-labeled-data conditions, indicating superior sample efficiency and robustness.
• Reward source and data scale are established as critical levers for open-domain agent learning; multi-role co-evolution enables scalable, self-improving agents.

Key Takeaways for Search / Rec / Personalization

1. Architecture: agentic paradigms are making retrieval frequency and strategy themselves learnable objects; 2. Data: synthetic data works at scale but needs safeguards against distribution shift; 3. Evaluation: GRM verification capacity is the bottleneck — reward models must keep training on evolving data distributions; 4. Engineering: latency, cost, interpretability, and safety remain hard constraints for real deployments.

Original Abstract (preserved)

> We study whether self-learning can scale LLM-based agents without relying on human-curated datasets or predefined rule-based rewards. Through controlled experiments in a search-agent setting, we identify two key determinants of scalable agent training: the source of reward signals and the scale of agent task data. We find that rewards from a Generative Reward Model (GRM) outperform rigid rule-based signals for open-domain learning, and that co-evolving the GRM with the policy further boosts performance. Increasing the volume of agent task data-even when synthetically generated-substantially enhances agentic capabilities. Building on these insights, we propose Agentic Self-Learning (ASL), a fully closed-loop, multi-role reinforcement learning framework that unifies task generation, policy execution, and evaluation within a shared tool environment and LLM backbone. ASL coordinates a Prompt Generator, a Policy Model, and a Generative Reward Model to form a virtuous cycle of harder task setting, sharper verification, and stronger solving. Empirically, ASL delivers steady, round-over-round gains, surpasses strong RLVR baselines (e.g., Search-R1) that plateau or degrade, and continues improving under zero-labeled-data conditions, indicating superior sample efficiency and robustness. We further show that GRM verification capacity is the main bottleneck: if frozen, it induces reward hacking and stalls progress; continual GRM training on the evolving data distribution mitigates this, and a small late-stage injection of real verification data raises the performance ceiling. This work establishes reward source and data scale as critical levers for open-domain agent learning and demonstrates the efficacy of multi-role co-evolution for scalable, self-improving agents. The data and code of this paper are released at https://github.com/forangel2014/Towards-Agentic-Self-Learning

References

• Paper: Towards Agentic Self-Learning LLMs in Search Environment — https://arxiv.org/abs/2510.14253
• Code and data: https://github.com/forangel2014/Towards-Agentic-Self-Learning