Towards Agentic Self-Learning LLMs in Search Environment

• Paper: arXiv:2510.14253
• Authors: Wangtao Sun, Xiang Cheng, Jialin Fan, Yao Xu, Xing Yu, Shizhu He, et al. (8 authors total)
• Published: 2025-10-16
• Code/Data: https://github.com/forangel2014/Towards-Agentic-Self-Learning
• Category: Agentic Search

Key points

• Studies whether self-learning can scale LLM-based search agents without human-curated datasets or predefined rule-based rewards.
• Identifies reward signal source and agent task data scale as the two critical determinants of scalable agent training.
• Generative Reward Model (GRM) rewards outperform rigid rule-based signals for open-domain learning; co-evolving the GRM with the policy further improves results.
• More agent task data — even fully synthetic — substantially enhances agentic capabilities.
• Proposes Agentic Self-Learning (ASL): a fully closed-loop, multi-role RL framework unifying task generation, policy execution, and evaluation on a shared tool environment and LLM backbone.

Method

ASL coordinates three roles within one framework:

1. Prompt Generator — creates increasingly harder tasks. 2. Policy Model — executes search-agent tasks via tool calls in the shared environment. 3. Generative Reward Model — verifies outcomes and provides learning signals.

These roles form a virtuous cycle: harder task setting, sharper verification, and stronger solving. The training pipeline follows the standard formalize → design → train → infer structure, with the twist that task generation and evaluation are themselves learned components co-evolving with the policy.

Results

• ASL delivers steady, round-over-round gains, whereas strong RLVR baselines (e.g., Search-R1) plateau or degrade.
• Continues improving under zero-labeled-data conditions, indicating superior sample efficiency and robustness.
• GRM verification capacity is the main bottleneck: if the GRM is frozen, reward hacking occurs and progress stalls; continual GRM training on the evolving data distribution mitigates this, and a small late-stage injection of real verification data raises the performance ceiling.

Implications for search/recommendation systems

• Architecture: cascaded retrieval + reranking + generation remains mainstream, but the agentic paradigm makes "when and how often to retrieve" itself a learnable policy.
• Data: synthetic data is valuable but requires attention to knowledge leakage and distribution shift.
• Evaluation: offline metrics increasingly diverge from online satisfaction; LLM-as-judge should be cross-validated with human evaluation.
• Productization: latency, cost, interpretability, and safety are hard constraints for real-world deployment.

Limitations and future work

Potential limitations include experiment scale bounded by compute budgets, benchmarks that may not match real user distributions, English-centric data leaving cross-lingual generalization unknown, and safety risks of open-web agent operation. Future directions include more efficient test-time compute allocation, deeper integration with knowledge graphs and structured databases, and causal/fairness constraints for recommendation scenarios.

Original abstract

> We study whether self-learning can scale LLM-based agents without relying on human-curated datasets or predefined rule-based rewards. Through controlled experiments in a search-agent setting, we identify two key determinants of scalable agent training: the source of reward signals and the scale of agent task data. We find that rewards from a Generative Reward Model (GRM) outperform rigid rule-based signals for open-domain learning, and that co-evolving the GRM with the policy further boosts performance. Increasing the volume of agent task data—even when synthetically generated—substantially enhances agentic capabilities. Building on these insights, we propose Agentic Self-Learning (ASL), a fully closed-loop, multi-role reinforcement learning framework that unifies task generation, policy execution, and evaluation within a shared tool environment and LLM backbone. ASL coordinates a Prompt Generator, a Policy Model, and a Generative Reward Model to form a virtuous cycle of harder task setting, sharper verification, and stronger solving. Empirically, ASL delivers steady, round-over-round gains, surpasses strong RLVR baselines (e.g., Search-R1) that plateau or degrade, and continues improving under zero-labeled-data conditions, indicating superior sample efficiency and robustness. We further show that GRM verification capacity is the main bottleneck: if frozen, it induces reward hacking and stalls progress; continual GRM training on the evolving data distribution mitigates this, and a small late-stage injection of real verification data raises the performance ceiling. This work establishes reward source and data scale as critical levers for open-domain agent learning and demonstrates the efficacy of multi-role co-evolution for scalable, self-improving agents. The data and code of this paper are released at https://github.com/forangel2014/Towards-Agentic-Self-Learning