Evolving-RL: A Single-Model Co-Evolutionary RL Framework for Self-Evolving Agents

> Paper: *Evolving-RL: End-to-End Optimization of Experience-Driven Self-Evolving Capability within Agents* > Authors: Zhiyuan Fan, Wenwei Jin, Feng Zhang, Bin Li, Yihong Dong (Peking University), Yao Hu, Jiawei Li (Xiaohongshu Inc.) > Paper: https://arxiv.org/abs/2605.10663 > Code: https://github.com/Fanzy27/Evolving-RL

Key points

• Problem: LLM agents are static after training. Existing "experience-driven self-evolving" methods only optimize skill storage/retrieval, not the model's ability to *extract* useful experience. Low-quality experience causes the model to learn to ignore all skills — "skill amnesia."
• Core insight: Experience extraction and utilization are two sides of the same problem and must co-evolve within one model.

Framework

Evolving-RL instantiates experience as compact text skills: what to do (decision rules), when to trigger (conditions), and how to recover from failure.

A single policy model π_θ plays two roles:

• Extractor: given a source-task trajectory, generates N=8 candidate skills.
• Solver: solves new tasks with skills injected in context.

Four-phase training loop

1. Solve source tasks without skills; collect trajectories and rewards. 2. Extract N=8 candidate skills per source task. 3. Downstream evaluation: retrieve K=4 semantically related tasks (embedding similarity); score each skill by average reward across those tasks. 4. Joint optimization with GRPO:

• Extractor advantage groups: N skills from the same source task; rewards come only from downstream tasks, forcing transferable (not source-overfit) skills.
• Solver advantage groups: N trajectories for the same task under different skills, teaching it to exploit good skills and resist bad ones.

Total loss: L = λ_e * L_e + λ_s * L_s applied to the same parameters, creating a mutually reinforcing loop. Stability measures: negative entropy regularization -η_e * H(π_θ) (extraction is naturally high-entropy) and rule-based filtering of skills with anomalous characters.

Results

ALFWorld (success rate):

| Method | Seen | Unseen | Overall | |---|---|---|---| | Base model | 51.9 | 27.4 | 45.5 | | GRPO | 96.2 | 33.7 | 79.9 | | GRPO (+skills) | 97.0 | 44.6 | 83.3 | | SkillRL | 86.2 | 68.8 | 81.7 | | Evolving-RL (no skills) | 97.4 | 81.1 | 93.1 | | Evolving-RL (+skills) | 98.6 | 88.6 | 96.0 |

• ~98.7% relative improvement on unseen tasks vs GRPO(+skills).
• Even without skill injection it far exceeds GRPO (81.1% vs 33.7%): experience patterns are internalized into parameters.
• Extracted skills transfer across models: base model 45.5% → 60.4%; GRPO model 79.9% → 88.8%.

Mind2Web (action accuracy): base 8.79 → GRPO 22.83 → Evolving-RL 28.05 (no skills) → 30.87 (with skills), a 35.8% relative gain over GRPO.

Ablations: extractor-only training overfits seen tasks; solver-only training becomes strong (90.9%) but gains nothing from injected skills (90.4%) — it learns to "ignore" them. Only co-evolution improves both seen and unseen performance. Injecting *irrelevant* skills performs no worse than no skills, showing gains come from skill semantics, not prompt formatting.

Why it works

• Transfer-utility-based skill evaluation: skill quality is defined by downstream performance, not source-task performance.
• The solver-only setting is a local optimum: exposed to noisy skills, the model optimally ignores all of them. Co-evolution keeps injected skill quality rising, breaking this optimum.
• Evolving-RL doubles as an experience-augmented RL algorithm: repeated exposure to skill-conditioned contexts internalizes reusable procedural patterns.

Relation to prior work

Unlike ExpeL, Memento, ReasoningBank (hand-crafted extraction, no utilization training), SkillRL (external extractor), and EvolveR (manual filtering), Evolving-RL is the only framework that RL-optimizes both extraction and utilization in a single shared model.

Limitations

• Simple skill management (embedding-based retrieval only).
• Noisy evaluation with small K=4; compute cost of N×K=32 downstream interactions per iteration (10–17 hours training).
• Only tested on ALFWorld and Mind2Web; deployment-time continual evolution, hierarchical skill libraries, cross-domain transfer, and multi-agent skill sharing remain open directions.

Takeaway

Evolving-RL turns experience from an add-on component into the soil in which agent capabilities grow: by co-evolving extraction and utilization in one model, the agent becomes an active producer of its own skills rather than a passive consumer of experience.

References

• Fan Z, Jin W, Zhang F, et al. Evolving-RL: End-to-End Optimization of Experience-Driven Self-Evolving Capability within Agents. arXiv:2605.10663, 2026.
• Shridhar M, Yuan X, et al. ALFWorld: Aligning Text and Embodied Environments. 2021.
• Deng X, et al. Mind2Web: Towards a Generalist Agent for the Web. 2023.
• Shao Z, et al. DeepSeekMath. 2024 (GRPO).