Towards Agentic Self-Learning LLMs in Search Environment

This is a Chinese forum summary of the paper Towards Agentic Self-Learning LLMs in Search Environment (arXiv:2510.14253, October 2025), by Wangtao Sun, Xiang Cheng, Jialin Fan, Yao Xu, Xing Yu, Shizhu He, et al. Code and data: https://github.com/forangel2014/Towards-Agentic-Self-Learning

Key points

• Research question: Can self-learning scale LLM-based agents without human-curated datasets or predefined rule-based rewards?
• Two key determinants of scalable agent training (identified via controlled experiments in a search-agent setting):
1. The source of reward signals — rewards from a Generative Reward Model (GRM) outperform rigid rule-based signals for open-domain learning. 2. The scale of agent task data — increasing task data volume substantially enhances agentic capabilities, even when the data is synthetically generated.
• Proposed framework — Agentic Self-Learning (ASL): a fully closed-loop, multi-role reinforcement learning framework that unifies task generation, policy execution, and evaluation within a shared tool environment and LLM backbone.
• Three coordinated roles: a Prompt Generator, a Policy Model, and a Generative Reward Model, forming a virtuous cycle of harder task setting, sharper verification, and stronger solving.

Empirical findings

• ASL delivers steady, round-over-round gains and surpasses strong RLVR baselines (e.g., Search-R1) that plateau or degrade.
• ASL continues improving under zero-labeled-data conditions, showing superior sample efficiency and robustness.
• GRM verification capacity is the main bottleneck: if the GRM is frozen, it induces reward hacking and stalls progress. Continual GRM training on the evolving data distribution mitigates this, and a small late-stage injection of real verification data raises the performance ceiling.

Context from the forum post

• The post situates the paper within agentic search and large-scale search/recommendation systems, noting that the LLM era introduces new variables of *reasoning budget* and *action space* (whether to retrieve, how many times, and which tools to call).
• Implications highlighted for Search/Rec/Personalization:
1. Architecture: cascade retrieval + rerank + generation remains mainstream, but the agentic paradigm makes retrieval strategy itself learnable. 2. Data: high-quality instruction data and click/session logs remain critical; synthetic data must guard against knowledge leakage and distribution shift. 3. Evaluation: the gap between offline metrics and online satisfaction is widening; LLM-as-judge needs cross-validation with human evaluation. 4. Product: latency, cost, interpretability, and safety are hard constraints for industrial deployment.
• Suggested limitations/future directions: experiment scale constrained by GPU budget, benchmark mismatch with real user distributions, unknown cross-lingual generalization from English-centric data, and open-web safety risks; future work may explore better test-time compute allocation and deeper integration with knowledge graphs.

Glossary (from the post)

| Term | Meaning | |------|---------| | RAG | Retrieval-Augmented Generation | | RLVR | Reinforcement Learning with Verifiable Rewards | | GRM | Generative Reward Model | | Agentic Search | Modeling search as sequential decision-making and tool calling |

Original abstract (verbatim)

> We study whether self-learning can scale LLM-based agents without relying on human-curated datasets or predefined rule-based rewards. Through controlled experiments in a search-agent setting, we identify two key determinants of scalable agent training: the source of reward signals and the scale of agent task data. We find that rewards from a Generative Reward Model (GRM) outperform rigid rule-based signals for open-domain learning, and that co-evolving the GRM with the policy further boosts performance. Increasing the volume of agent task data-even when synthetically generated-substantially enhances agentic capabilities. Building on these insights, we propose Agentic Self-Learning (ASL), a fully closed-loop, multi-role reinforcement learning framework that unifies task generation, policy execution, and evaluation within a shared tool environment and LLM backbone. ASL coordinates a Prompt Generator, a Policy Model, and a Generative Reward Model to form a virtuous cycle of harder task setting, sharper verification, and stronger solving. Empirically, ASL delivers steady, round-over-round gains, surpasses strong RLVR baselines (e.g., Search-R1) that plateau or degrade, and continues improving under zero-labeled-data conditions, indicating superior sample efficiency and robustness. We further show that GRM verification capacity is the main bottleneck: if frozen, it induces reward hacking and stalls progress; continual GRM training on the evolving data distribution mitigates this, and a small late-stage injection of real verification data raises the performance ceiling. This work establishes reward source and data scale as critical levers for open-domain agent learning and demonstrates the efficacy of multi-role co-evolution for scalable, self-improving agents. The data and code of this paper are released at https://github.com/forangel2014/Towards-Agentic-Self-Learning