Skill1 Deep Dive: How Meituan Makes Agent Skill Libraries Evolve Themselves with RL

Forum topic · 小凯 · 2026-05-15

Summary

A detailed analysis of Skill1 (Unified Evolution of Skill-Augmented Agents via Reinforcement Learning), a paper by Yaorui Shi et al. from Meituan's LongCat team (arXiv:2605.06130). Skill1 unifies three traditionally separate capabilities—skill selection, skill utilization, and skill distillation—into a single policy trained with one shared reward signal. The key insight is a temporal decomposition of the task outcome signal: a low-frequency trend (EMA) provides credit for skill selection, high-frequency residuals (performance above the current best skill's utility) trigger distillation of new skills, and the raw reward optimizes execution. On ALFWorld, Skill1 reaches 97.5% average success rate, beating RetroAgent (94.9%) and skill-free GiGPO (90.8%), with ablations showing -16.6 points without the skill library, -5.7 without selection, and -5.1 without distillation. Training curves reveal a positive feedback loop: selection accuracy converges first, then success rate and distillation improve together. The post also critiques limitations: a 5,000-skill capacity cap, a frozen retrieval encoder (all-MiniLM-L6-v2), no learning from failures, and a flat (non-hierarchical) skill store.

Paper: *Skill1: Unified Evolution of Skill-Augmented Agents via Reinforcement Learning* Authors: Yaorui Shi et al. (Meituan / LongCat team) arXiv: 2605.06130

One-Sentence Conclusion

A skill library is not an "external hard drive." If you treat selection, utilization, and distillation as three independent modules with separate objectives, your Agent is effectively doing three things that fight each other. Skill1's core insight: use one policy, one signal, and one set of gradients so that all three capabilities co-evolve around the single goal of task success.

The Problem: Why Skill Libraries Become Junk Drawers

Think of an assistant with a handbook of past experience. Three failure modes arise:

Selection: which page to consult for a new task — a wrong pick ruins everything downstream
Utilization: even the right skill can be executed poorly
Distillation: after a task, what gets written back? Raw transcripts, or reusable patterns?

Prior approaches either split these into modules with different reward signals, or freeze one or two and delegate them to external systems. The result: selection never learns, execution never improves, and distillation injects noise. Skill1 argues these are not a pipeline but a single policy's three-part performance across different time scales.

Core Idea: One Signal, Three Decompositions

Skill1 does not invent a new reward function; it decomposes the same task-outcome signal r(τ) in the time domain:

| Component | Frequency | Attribution | Intuition | |:---|:---|:---|:---| | Low-frequency trend | Slow | Skill selection | Consistently useful skills gain credit | | High-frequency change | Fast | Skill distillation | Experience that breaks the current ceiling becomes a new skill | | Absolute value | Raw | Skill utilization | Directly optimizes execution |

Implementation

Selection — exponential moving average:

U(s) \leftarrow (1-\alpha) \cdot U(s) + \alpha \cdot r(\tau_i)

Crucially, all retrieved candidates are updated, not just the chosen one — co-retrieval itself is evidence, granting indirect credit to unselected but consistently successful skills.

Distillation — residual:

R^{\text{distill}} = r(\tau_i) - \hat{U}_i

where $\hat{U}_i$ is the highest utility among retrieved candidates. Only experiences exceeding the library's current ceiling get distilled; worse ones are suppressed, preventing junk from entering the library.

Utilization:

R^{\text{util}} = r(\tau_i)

Unified Objective

J(\theta) = J^{\text{util}} + \lambda_1 J^{\text{rerank}} + \lambda_2 J^{\text{distill}}

Defaults: $\lambda_1 = \lambda_2 = 0.3$ . Utilization uses GRPO; reranking uses REINFORCE (candidates differ, so no in-group comparison); distillation uses independently normalized GRPO.

Results

ALFWorld (average success rate over 6 task types)

| Method | Avg Success | Gain | |:---|:---|:---| | Zero-Shot | 14.8% | — | | ReAct | 31.2% | — | | GRPO (no skills) | 77.6% | — | | GiGPO (best skill-free) | 90.8% | — | | RetroAgent (prior best) | 94.9% | — | | Skill1 | 97.5% | +2.6 |

First place in 5 of 6 task types (Clean: 97.3 vs 99.2, p=0.147, not significant)
Largest gain on Look (+7.7 points) — the task needing the most skill composition
+6.7 points over the best skill-free method (GiGPO)

WebShop

| Method | Score | Success | |:---|:---|:---| | GiGPO | 84.4 | 72.8 | | RetroAgent | 88.9 | 82.3 | | Skill1 | 89.7 | 82.9 |

Ablations

| Variant | Avg Success | Drop | |:---|:---|:---| | Skill1 (full) | 97.5% | — | | w/o selection | 91.8% | -5.7 | | w/o distillation | 92.4% | -5.1 | | w/o skill library | 80.9% | -16.6 | | w/o both (λ₁=λ₂=0) | 90.2% | -7.3 |

Removing the library hurts most — it is the foundation. Removing selection hurts multi-step routing tasks most; without distillation the library stores raw trajectories, making selection noisier. The two auxiliary objectives are complementary.

Training Dynamics: Genuine Co-Evolution

| Metric | Convergence | Order | |:---|:---|:---| | Selection accuracy | 0.95 @ step 20 | First | | Utilization success rate | 0.8 @ step 60 | Second | | Distillation positive rate | 0.8 @ step 60 | Synchronized | | Top-skill utility | 0.91 @ step 85 | Last |

A clear causal chain: selection converges first → better skill supply raises success rate → high-quality trajectories feed distillation → new skills raise the library's ceiling, improving selection options. A positive feedback loop, not three separate training processes.

Strengths and Weaknesses

Strengths:

Avoids "cargo-cult" design: instead of keeping the form of a skill library with frozen or externally controlled components, Skill1 internalizes everything into one learning policy driven by a shared signal.
The training-dynamics figures demonstrate co-evolution more convincingly than theory could.

Limitations acknowledged by the authors:

| Limitation | Detail | |:---|:---| | Library capacity | 5,000 skills max; eviction by utility × usage frequency | | Frozen encoder | Retrieval uses fixed all-MiniLM-L6-v2 embeddings | | Single task signal | Assumes task success is a sufficient statistic; ignores intermediate feedback | | Skill representation | Text-only policies and descriptions; no structured representations |

Additional questions raised in the post: 1. The EMA rate α=0.05 is manually set — is the optimum task-dependent? 2. Distillation triggers only when r(τ)=1 — can "negative skills" (anti-patterns) be learned from failures? 3. The library is flat — does retrieving among 5,000 peer skills cause "choice overload"?

Future Directions

1. Hierarchical skill libraries: tree-structured retrieval from coarse to fine to reduce selection noise 2. Negative skill distillation: extracting anti-patterns from failures for robustness 3. Learnable retrieval encoders: evolving skill embeddings rather than frozen ones 4. Cross-task transfer: measuring skill reuse across domains (ALFWorld → WebShop) 5. Relation to HeavySkill: unifying Meituan's concurrent HeavySkill (collective deliberation) and Skill1 (skill evolution) into one framework

References

Paper: https://arxiv.org/abs/2605.06130
Related prior work (Meituan HeavySkill): https://zhichai.net/t/177620055