Towards Agentic Self-Learning LLMs in Search Environment

• Paper: https://arxiv.org/abs/2510.14253
• Code/Data: https://github.com/forangel2014/Towards-Agentic-Self-Learning
• Authors: Wangtao Sun, Xiang Cheng, Jialin Fan, Yao Xu, Xing Yu, Shizhu He, et al. (8 authors)
• Published: 2025-10-16

Key points

• Studies whether LLM-based search agents can be trained via self-learning without human-curated datasets or predefined rule-based rewards.
• Identifies two critical levers for scalable agent training: reward signal source and scale of agent task data.
• Finds that rewards from a Generative Reward Model (GRM) outperform rigid rule-based signals for open-domain learning, and co-evolving the GRM with the policy further improves results.
• Shows that increasing agent task data volume — even synthetic — substantially enhances agentic capabilities.
• Proposes Agentic Self-Learning (ASL), a fully closed-loop, multi-role reinforcement learning framework.
• Demonstrates steady round-over-round gains, surpassing RLVR baselines (e.g., Search-R1) that plateau or degrade, with continued improvement under zero-labeled-data conditions.
• Identifies GRM verification capacity as the main bottleneck: a frozen GRM induces reward hacking; continual GRM training on evolving data mitigates this, and a small late-stage injection of real verification data raises the performance ceiling.

The ASL Framework

ASL unifies task generation, policy execution, and evaluation within a shared tool environment and LLM backbone, coordinating three roles:

1. Prompt Generator — creates progressively harder tasks. 2. Policy Model — solves search-agent tasks via tool use. 3. Generative Reward Model — verifies solutions and provides rewards.

Together these form a virtuous cycle: harder task setting, sharper verification, and stronger solving.

Experimental findings

• ASL delivers steady, round-over-round improvement, whereas strong RLVR baselines (e.g., Search-R1) plateau or degrade.
• Learning continues even with zero labeled data, indicating superior sample efficiency and robustness.
• GRM verification capacity is the bottleneck: freezing the GRM stalls progress via reward hacking; continual GRM training plus a small amount of late-stage real verification data raises the ceiling.

Implications for search and recommendation systems

• Architecture: agentic paradigms make retrieval count and strategy themselves learnable objects, beyond the classic cascade of retrieval, re-ranking, and generation.
• Data: synthetic data works at scale but requires safeguards against knowledge leakage and distribution shift.
• Evaluation: LLM-as-judge signals need cross-validation with human assessment; process metrics (task success, citation accuracy) matter more than static nDCG.
• Production: latency, cost, interpretability, and safety remain hard constraints for industrial deployment.

The authors establish reward source and data scale as critical levers for open-domain agent learning and demonstrate the efficacy of multi-role co-evolution for scalable, self-improving agents.