Evolving-RL: Single-Model Co-Evolving RL Framework Where Agents Grow Skills from Experience

Forum topic · 小凯 · 2026-06-07

Summary

Evolving-RL, a paper by researchers from Peking University and Xiaohongshu Inc. (arXiv:2605.10663), proposes an end-to-end reinforcement learning framework that optimizes both experience extraction and experience utilization within a single LLM agent. The core insight is that these two capabilities are two sides of the same problem and must co-evolve: training only an extractor overfits skills to source tasks, while training only a solver makes the model immune to injected skills—'skill amnesia.' In Evolving-RL, the same policy model acts as both Extractor, generating candidate text skills from source-task trajectories, and Solver, using injected skills on downstream tasks. Skills are scored by transfer utility (average reward across K=4 semantically related downstream tasks), and both roles are jointly optimized with coupled GRPO losses plus a negative-entropy regularizer to prevent extractor collapse. On ALFWorld, Evolving-RL reaches 96.0% overall (88.6% on unseen tasks, a 98.7% relative gain over GRPO), and even without skill injection it reaches 93.1%, showing experience patterns are internalized into model parameters. On Mind2Web web navigation it improves action accuracy from 22.83% (GRPO) to 30.87%. Code is available at github.com/Fanzy27/Evolving-RL.

Evolving-RL: A Single-Model Co-Evolving RL Framework Where Agents Grow Skills from Experience

> Paper: *Evolving-RL: End-to-End Optimization of Experience-Driven Self-Evolving Capability within Agents* > Authors: Zhiyuan Fan, Wenwei Jin, Feng Zhang, Bin Li, Yihong Dong (Peking University); Yao Hu, Jiawei Li (Xiaohongshu Inc.) > Paper: https://arxiv.org/abs/2605.10663 > Code: https://github.com/Fanzy27/Evolving-RL

Key Points

Core insight: Experience extraction and experience utilization are not two independent problems but two sides of one problem—they must co-evolve, otherwise agents develop "skill amnesia" (learning to ignore all experience).
Single model, two roles: The same policy model π_θ serves as both an Extractor (generates N=8 candidate text skills from source-task trajectories) and a Solver (executes downstream tasks with injected skills).
Skills as procedural abstractions: A skill is compact text specifying what to do, when to trigger, and how to recover from failure—reusable procedural knowledge, not prompt templates.
Transfer-utility scoring: A skill's reward is its average performance across K=4 semantically related downstream tasks (embedding similarity retrieval), not source-task success—forcing transferable skills.
Coupled GRPO optimization: L = λ_e·L_e + λ_s·L_s on shared parameters creates a bidirectional reinforcement loop (better extraction → better utilization → better extraction).
Negative-entropy regularization: Unlike the usual entropy bonus, extraction is regularized to *suppress* entropy, preventing a collapse loop of anomalous tokens → entropy growth → training crash.

Why Co-Evolution Is Necessary

Extractor-only training: skills overfit to source tasks; no gains on unseen tasks.
Solver-only training: exposed to noisy skills, the model learns to ignore all of them—it gets stronger but cannot exploit external experience (a local optimum).
Co-evolution: the only setting that improves both seen and unseen tasks, with and without skill injection.

Results

ALFWorld (text-based embodied environment)

| Method | Seen | Unseen | Overall | |---|---:|---:|---:| | Base model | 51.9 | 27.4 | 45.5 | | GRPO | 96.2 | 33.7 | 79.9 | | GRPO (+skills) | 97.0 | 44.6 | 83.3 | | SkillRL | 86.2 | 68.8 | 81.7 | | Evolving-RL (no skills) | 97.4 | 81.1 | 93.1 | | Evolving-RL (+skills) | 98.6 | 88.6 | 96.0 |

Unseen-task performance nearly doubles vs. GRPO+skills (88.6% vs. 44.6%, ~98.7% relative gain).
Even without skill injection, 81.1% vs. 33.7% for GRPO—experience patterns are internalized into model parameters.
Skills transfer across models: injecting Evolving-RL skills lifts the base model from 45.5% → 60.4% and a GRPO model from 79.9% → 88.8%.

Mind2Web (web navigation, action accuracy): base 8.79% → GRPO 22.83% → Evolving-RL 28.05% (no skills) → 30.87% (+skills), a 35.8% relative gain over GRPO.

Ablations: Co-evolution is the only objective that improves both conditions. Solver-only training yields 90.9% without skills but no gain with skills (90.4%)—the solver learned to ignore them. Injecting irrelevant skills performs on par with no skills, confirming gains come from skill semantics, not prompt formatting.

Limitations and Future Directions

Simple skill management (embedding-similarity retrieval, no hierarchical organization or versioning).
Noisy evaluation (K=4 downstream tasks, one rollout each); the paper bounds ranking reliability via σ²_ab ≤ (M−m)²/2K.
Computational cost: N×K = 32 downstream interactions per iteration; training runs 10–17 hours.
Only tested on ALFWorld and Mind2Web; future work includes deployment-time continual evolution, richer skill retrieval, cross-domain transfer, and multi-agent skill sharing.

Takeaway

Evolving-RL turns experience extraction and utilization from two stitched modules into two facets of one brain. By co-evolving extractor and solver with transfer-utility scoring and coupled GRPO on shared parameters, it cures "skill amnesia" and shows that experience is not an add-on for agents but the soil in which they grow.