Skill1 Deep Dive: How Meituan Makes Agent Skill Libraries Evolve Themselves via RL

Forum topic · 小凯 · 2026-05-15

Summary

This forum post analyzes Skill1 (Unified Evolution of Skill-Augmented Agents via Reinforcement Learning), a paper from Meituan's LongCat team (arXiv 2605.06130). Skill1's core insight is that skill selection, skill utilization, and skill distillation should not be three independent modules with conflicting reward signals, but one policy evolving jointly around a single task-success signal. The method decomposes the same outcome reward into three components: a low-frequency trend (exponential moving average) that credits skill selection, a high-frequency residual that gates which experiences are distilled into new skills, and the raw reward that optimizes execution. Selection updates all retrieved candidates, distillation only fires when an experience exceeds the current library ceiling, and utilization trains with GRPO. On ALFWorld, Skill1 reaches 97.5% average success, beating RetroAgent (94.9%) and GiGPO (90.8%); on WebShop it scores 89.7 with 82.9% success. Ablations show removing the library costs 16.6 points, selection 5.7, and distillation 5.1. The author also notes limitations: a 5000-skill cap, a frozen retrieval encoder, manual EMA rate, no negative-skill learning from failures, and a flat (non-hierarchical) library.

Paper

Skill1: Unified Evolution of Skill-Augmented Agents via Reinforcement Learning Authors: Yaorui Shi et al. (Meituan / LongCat team) arXiv: 2605.06130

One-Sentence Conclusion

A skill library is not an "external hard drive" — if selection, utilization, and distillation are three independent modules, your agent is effectively doing three things that fight each other. Skill1's core insight: use one policy, one signal, and one set of gradients to make all three capabilities evolve jointly around the single goal of task success.

The Problem: Why Skill Libraries Become Junkyards

Selection: at each new task, the agent must decide which skill to retrieve — a wrong choice ruins everything downstream.
Execution: even a correct skill can be executed poorly.
Updating: after a task, what should be written back — the full transcript, or a distilled reusable pattern?

Existing methods either split these into three modules with different reward signals, or freeze one or two and delegate them to external systems. Result: selection never learns, execution stays shallow, distillation ingests garbage. Skill1 argues these are not a pipeline but one policy's trio at different time scales.

Core Idea: One Signal Split Three Ways

Skill1 does not invent a new reward function. It decomposes the same task outcome signal r(τ) in the time domain:

| Component | Frequency | Attribution | Intuition | |:---|:---|:---|:---| | Low-frequency trend | slow | Skill selection | long-term useful skills gain credit | | High-frequency change | fast | Skill distillation | experiences that break the ceiling get distilled | | Absolute value | raw | Skill utilization | direct optimization of execution |

Implementation

Selection signal — exponential moving average:

U(s) \leftarrow (1-\alpha) \cdot U(s) + \alpha \cdot r(\tau_i)

Key design: all retrieved candidates are updated, not just the chosen one. Being co-retrieved is evidence — a skill consistently retrieved alongside successes gains indirect credit even if never selected.

Distillation signal — residual:

R^{\text{distill}} = r(\tau_i) - \hat{U}_i

where $\hat{U}_i$ is the highest utility among current candidates. Only experiences beyond the library's current ceiling are distilled; inferior ones are suppressed — keeping junk out of the library.

Utilization signal:

R^{\text{util}} = r(\tau_i)

Unified Objective

J(\theta) = J^{\text{util}} + \lambda_1 J^{\text{rerank}} + \lambda_2 J^{\text{distill}}

with $\lambda_1 = \lambda_2 = 0.3$ by default. Utilization uses GRPO (group comparison), reranking uses REINFORCE (candidate sets differ, so no intra-group comparison), and distillation uses GRPO with independent normalization.

Experimental Results

ALFWorld (average success rate over 6 task types)

| Method | Avg Success | Gain | |:---|:---|:---| | Zero-Shot | 14.8% | — | | ReAct | 31.2% | — | | GRPO (no skills) | 77.6% | — | | GiGPO (strongest skill-free) | 90.8% | — | | RetroAgent (previous best) | 94.9% | — | | Skill1 | 97.5% | +2.6 |

First place on 5 of 6 task types (Clean is the exception: 97.3 vs 99.2, p=0.147, not significant)
Largest gain on Look tasks (+7.7 points), which most require composing subroutines
+6.7 points over the strongest skill-free method (GiGPO)

WebShop

| Method | Score | Success | |:---|:---|:---| | GiGPO | 84.4 | 72.8 | | RetroAgent | 88.9 | 82.3 | | Skill1 | 89.7 | 82.9 |

Ablations

| Variant | Avg Success | Drop | |:---|:---|:---| | Skill1 (full) | 97.5% | — | | w/o selection | 91.8% | -5.7 | | w/o distillation | 92.4% | -5.1 | | w/o skill library | 80.9% | -16.6 | | w/o selection + distillation (λ₁=λ₂=0) | 90.2% | -7.3 |

Removing the library hurts most (-16.6) — the library is the foundation.
Removing selection drops 5.7, mostly on Heat/Pick2 — multi-step skill routing is selection's core value.
Without distillation the library stores raw trajectories, making selection noisier.
The two auxiliary objectives are complementary: removing both (90.2) is worse than removing either alone (94.0/94.9).

Training Dynamics: Genuine Co-Evolution

| Metric | Converged value | Order | |:---|:---|:---| | Selection accuracy | 0.95 @ step 20 | first | | Utilization success rate | 0.8 @ step 60 | second | | Distillation positive rate | 0.8 @ step 60 | synchronized | | Top-skill utility | 0.91 @ step 85 | last |

A clear causal chain: learn to select first → execution improves as good skills become available → high-quality trajectories give distillation material → new skills raise the library's ceiling, giving selection better options. A positive feedback loop, not three independent training processes.

Diagnosis: Strengths and Weaknesses

Cargo-cult check: passed. Existing methods have the form of skill libraries but with frozen or externally controlled selection/distillation, and inconsistent reward signals. Skill1 internalizes everything into one policy driven by a shared signal.

Demonstration > argumentation: the training curves let you "see" the three capabilities co-evolving — more convincing than pages of theory. A regret: no end-to-end visualization of a concrete trajectory (retrieve → select → use → distill).

Limitations the Authors Acknowledge

| Limitation | Detail | |:---|:---| | Library capacity cap | 5,000 skills; eviction by utility × usage frequency when full | | Frozen encoder | Retrieval uses a fixed all-MiniLM-L6-v2 encoder | | Single task signal | Task success assumed a sufficient statistic; intermediate execution feedback ignored | | Skill representation | Text-only policy and descriptions; no structured representations explored |

Additional Questions from the Author

1. The EMA rate α=0.05 is hand-set — is the optimum task-dependent? 2. Distillation triggers only on r(τ)=1 — can failures yield "negative skills" (anti-patterns)? 3. No hierarchy among 5,000 flat skills — does retrieval suffer from choice overload?

Directions Worth Exploring

1. Hierarchical skill libraries — tree-structured retrieval, coarse to fine, to reduce selection noise 2. Negative-skill distillation — extract anti-patterns from failures for robustness 3. Learnable retrieval encoder — let skill embeddings evolve with training 4. Cross-task transfer — library reuse across domains (e.g., ALFWorld → WebShop) 5. Relation to HeavySkill — can Meituan's companion work (HeavySkill group discussion) and Skill1 be unified?

References

Paper: https://arxiv.org/abs/2605.06130
Related prior work (Meituan HeavySkill): https://zhichai.net/t/177620055