Native Evolution: Teaching LLM Agents to Self-Explore Without Rewards or Tasks

Paper Overview

Title: Training LLM Agents for Spontaneous, Reward-Free Self-Evolution via World Knowledge Exploration Authors: Qifan Zhang, Dongyang Ma, Tianqing Fang, et al. (Tencent AI Lab / HKUST(GZ)) Venue: arXiv:2604.18131v1 [cs.AI], 20 Apr 2026

Key Points

The problem with current "self-evolving" agents

• The paper opens with a bold claim: most "self-evolving" agents are an illusion.
• Existing approaches fall into two camps:
• Experience-driven evolution: humans design tasks and reward functions; agents passively solve them and accumulate experience that tunes prompts, memory, skills, or weights.
• Adversarial evolution: a challenger agent generates tasks for a solver agent, but the overall pipeline is still human-designed.
• Both paradigms require human guidance to function, do not let agents spontaneously explore environments, and do not build a model of the world.

The Native Evolution paradigm

• Native Evolution trains an agent with an intrinsic meta-evolution capability that works in two stages:
1. Native Evolution Phase: Upon entering an environment E, the agent autonomously explores and produces a structured World Knowledge document K, denoted π_evolve(K | E). There is no task, no reward, and no human intervention at inference time. 2. Knowledge-Enhanced Execution Phase: When a task arrives, the agent uses K to guide action selection: a_t ~ π_task(a_t | o_t, K, Task).
• World Knowledge K is a compact, Markdown-formatted representation of the specific environment instance, distinct from reusable skills because it captures the intrinsic logic of that particular environment.

Training methodology

• A core challenge is that exploration is task-agnostic, so there is no ground-truth K. The solution is an outcome-based reward that defines K's quality by its downstream utility: R_evolve(K) = Success(T_E | K) - Success(T_E | ∅).
• The reward is used only during training; inference is fully autonomous.
• Training set: 600 deep-search questions across 20 domains of websites.
• Stage 1 (SFT): Gemini-2.5-Pro acts as a teacher, autonomously exploring diverse web environments, generating 3 candidate K's per environment, filtering with the reward, and using the best K* and its full trajectory (~374.8 steps, ~3,322.4 tokens per step) to fine-tune the base model. Gemini's K* raised Qwen3-30B-A3B accuracy by an average of 10.72 percentage points.
• Stage 2 (RFT, rejection-sampling fine-tuning): Standard RL is avoided because the generation horizon is hundreds of steps deep, rewards are sparse, and reward computation is expensive. Instead, the SFT model rolls out C candidates, scores them with R_evolve, retains high-scoring trajectories, and iterates for 2 rounds to obtain the final policy π_θ*.

Experimental results

• Evaluation uses WebWalker (4 domains: Conference, Game, Organization, Education; 10 sites each) and WebVoyager (4 sites: Wolfram, Apple, Dictionary, Coursera), for 1,427 samples after filtering out questions answerable from pretraining knowledge.
• On WebWalker, Qwen3-30B-A3B improves from a baseline of 26–42% to 36–58% depending on the domain, with the RFT variant adding a few extra points over the SFT variant. Seed-OSS-36B reaches 42–62% with the RFT variant.
• The most striking finding: a compact 14B Qwen3 model equipped with generated K outperforms unassisted Gemini-2.5-Flash, suggesting that environmental knowledge can compensate for scale.

Additional findings

• Transferability (RQ2): K generated by one model helps other models perform downstream tasks, implying K captures the objective structure of the environment rather than model-specific artifacts.
• SFT vs RFT (RQ3): SFT provides the basic exploration instinct; RFT refines the exploration strategy and produces more compact, higher-utility K.
• Length of K (RQ4): Under a token budget, information density matters more than length. Effective K encodes the environment's critical structure within a limited token budget.

Conceptual implications

• The shift is from reactive task-solving to proactive environment understanding, analogous to a local resident (who studies the city first) versus a temporary worker (who only shows up when paid).
• K is not pretrained general knowledge, not a task-specific skill, and not a simple page summary; it is a cognitive map of a specific environment instance that enables cross-task transfer.
• The outcome-based reward teaches the agent how to learn rather than what to learn, similar to how a coach trains the skill of shooting rather than handing over a finished free-throw routine.
• Unlike Test-Time Training, Native Evolution does not modify model weights; it only generates an external knowledge artifact that is injected into the prompt, making it compatible with high-throughput inference.

Limitations and open questions

• Training cost: generating high-quality teacher trajectories with Gemini-2.5-Pro and evaluating K on multiple downstream tasks is far more expensive than standard SFT.
• Environment assumptions: the approach assumes environments are static or slowly evolving, are explorable, and have acceptable exploration cost (≈374 steps × ~3,322 tokens per step is heavy). Highly dynamic or hostile environments may not fit.
• "Reward-free" is bounded: although inference is reward-free, the agent's exploration preferences are shaped by training data and the outcome-based reward, so this is more accurately described as "no explicit human intervention at inference time."
• Common objections: the approach is not identical to RAG (K is actively generated, not retrieved) and is not just prompt engineering (it is learned behavior); the 14B-vs-Gemini-2.5-Flash comparison is "14B + K" versus "Gemini without K," but the result still demonstrates a real enabling effect of K.

Outlook for the agent ecosystem

• The paper signals a shift in goal: the next generation of agents should be better environment understanders, not just better task solvers.
• Native Evolution is complementary to skill-orchestration methods like GraSP: first build an environment-level K, then compose K with skills into a DAG for complex tasks, finally executing with verification and local repair.
• For skill-based platforms such as OpenClaw, the paradigm suggests adding an environment-exploration mode on entering a new channel or service, compressing discoveries into a channel knowledge document, and reusing it across all later tasks.

Key quotes from the paper

• "Current research on 'self-evolving' agents is largely an illusion. Most existing methods do not allow an agent to evolve on its own; instead, they depend on human-defined workflows and verified reward signals to guide every step of improvement."
• "Human intelligence is naturally curious and proactive. When we enter a new city or start using a new software, we spontaneously learn the layout and the underlying logic, even without a specific task or a verified reward."
• "The quality of World Knowledge K is defined by its end-to-end utility—specifically, how much it 'empowers' the agent to perform better in that environment."
• "At inference time, the agent requires no external rewards or human instructions. It spontaneously performs native self-evolution to adapt to unknown environments using its internal parameters."

One-sentence takeaway

> Native Evolution upgrades agents from task-driven temporary workers into environment-aware local residents. Training teaches the agent how to autonomously explore and compress an environment into a reusable knowledge map; at inference, the agent builds that map spontaneously, then uses it to solve downstream tasks efficiently—showing that understanding the environment can outweigh raw parameter count.

References

• Zhang, Q., Ma, D., Fang, T., et al. (2026). Training LLM Agents for Spontaneous, Reward-Free Self-Evolution via World Knowledge Exploration. arXiv:2604.18131v1.