When AI Learns to Write Operating Manuals — Two Philosophies of Skill Self-Evolution
> Paper 1: EmbodiSkill (arXiv:2605.10332) — Nanjing University, Tsinghua AIR, Microsoft Research, et al. > Paper 2: SKILLEVOLVER (arXiv:2605.10500) — Tsinghua University, Beijing Jiaotong University
1. What Is a "Skill"?
Both papers discuss "skills," but the term is heavily overloaded in AI. A shared core idea: a skill is not memory — it is a reusable operating manual.
- In EmbodiSkill, a skill is a persistent, modifiable procedural specification — "find the key before opening the door," not "remember you tripped here last time."
- In SKILLEVOLVER, a skill is a portable artifact — prose instructions + executable code + examples, packaged as a directory that any CLI agent can load.
- ALFWorld: Qwen3.5-27B + evolved skills → 93.28% (beats GPT-5.2 direct execution by 31.58 points)
- Beats G-Memory (strongest memory baseline) by 25.01%
- Skill-aware vs. skill-unaware: 19.04% relative gain
- SkillsBench (83 tasks): 56.8% vs. 43.6% human-curated vs. 29.9% no-skill
- KernelBench GPU optimization: mean speedup 1.16 → 1.51
- Refinement (R=2 vs. R=1) contributes about 2/3 of total gains
- Downstream efficiency: −19.4% tokens, −15.3% interaction turns, −23.8% wall-clock
- Ju, R., et al. (2026). EmbodiSkill: Skill-Aware Reflection for Self-Evolving Embodied Agents. arXiv:2605.10332.
- Zhang, G., et al. (2026). SKILLEVOLVER: Skill Learning as a Meta-Skill. arXiv:2605.10500.
- Li, Y., et al. (2026). SkillsBench: A Benchmark for Skill-Centric Agent Evaluation.
- Ni, Y., et al. (2026). Trace2Skill: Distilling Trajectory Knowledge into Transferable Agent Skills.
- Xia, Y., et al. (2026). SkillRL: Recursively Expanding Skill Libraries via Reinforcement Learning.
Memory is "what I did last time"; a skill is "what should be done next time." Skills let AI extract general rules from past experience and reuse them in similar-but-different scenarios.
2. Why "No Weight Updates" Is Revolutionary
Both papers share a striking consensus: model weights never change. EmbodiSkill freezes the Qwen3.5-27B executor; SKILLEVOLVER's meta-skill never touches Claude Opus's parameters. All improvements go into external skill files.
This is a deliberate design choice, not laziness. Improvements stored in skill files are: 1. Version-controllable — watch skills evolve via git diff 2. Human-auditable — open the files and see what the AI learned 3. Cross-model reusable — skills learned by Qwen can be used by GPT 4. Rollback-able — revert a broken skill to a previous version
Reality check: this is an elegant choice under constraints — for closed-source APIs, edge devices, or enterprise compliance where weight updates are impossible or expensive, external skills are the only viable evolution path.
3. Two Philosophies of Learning from Failure
EmbodiSkill: First Determine Whose Fault It Is
Embodied agents face a problem that does not exist in CLI/chat: when a task fails, is the skill wrong or the execution wrong? EmbodiSkill's key contribution is four reflection types that force the AI to answer "whose fault is it?" before modifying anything:
| Reflection type | Meaning | Action | |---|---|---| | DISCOVERY | Trajectory reveals new knowledge not covered by the skill | Add new content to the skill body | | OPTIMIZATION | Skill works, but trajectory shows a better way | Modify target skill content | | SKILL DEFECT | Skill content is wrong, incomplete, or underspecified | Correct target skill content | | EXECUTION LAPSE | Skill is correct, but the agent failed to follow it | Add a reminder to the skill appendix |
The standout is EXECUTION LAPSE: traditional methods rewrite correct skills after every failure, diluting and polluting them. EmbodiSkill distinguishes "the skill is wrong" from "the agent didn't follow it." Effect: on ALFWorld, the skill-aware version improves over the skill-unaware version by a relative 19.04%.
SKILLEVOLVER: Learn from Deployment, Not Exploration Distillation
SKILLEVOLVER's insight: a skill's real test is whether another agent can use it as a dependency. Prior methods (e.g., Trace2Skill) distill skills from exploration trajectories, which do not reflect real deployment.
Process: 1. Write a candidate skill 2. Deploy it to fresh agents (Domain-Skill Agents) 3. Observe where they get stuck 4. Learn by contrasting high-reward vs. low-reward trajectories 5. Have an independent Auditor check for hardcoding, non-portability, and silent bypassing
Learning signals thus come from deployment failures, not exploration failures — exposing the author's blind spots (steps omitted as "common sense"). The meta-skill also performs strategy-diversified exploration: explicitly writing out different solution strategies (different libraries, algorithms, approaches) rather than just raising sampling temperature.
4. Honest Technical Comparison
| Dimension | EmbodiSkill | SKILLEVOLVER | |---|---|---| | Target environment | Physical/3D embodied (ALFWorld, EmbodiedBench) | CLI/code (SkillsBench, KernelBench) | | Skill carrier | Skill body + skill appendix | Portable directory: prose + code + examples | | Learning signal | Single-trajectory introspection (4 reflection types) | Multi-trajectory contrast (high vs. low reward) | | Evolution driver | Trajectory analysis → targeted edits | Deployment observation → contrastive updates → audit | | Failure classification | 4 reflection types (incl. execution lapses) | Audit checklist (overfitting, hardcoding, silent bypass, etc.) | | Verification | Skill appendix reinforces correct content | Independent Auditor with fresh sessions | | Per-task cost | Not clearly reported | ~$4/task (incl. one +8% refinement) | | Portability | Skill body tied to a specific executor | Artifacts loadable into any CLI agent |
5. The Numbers
EmbodiSkill
Takeaway: an open-weights local model with external skill evolution can outperform the strongest closed model executing directly — externalizing procedural knowledge can matter more than parameter scale.
SKILLEVOLVER
Takeaways: a single exploration pass (48.2%) already beats human skills (43.6%), but refinement extends the gap to +13.3pp — the write→deploy→observe→fix loop is a qualitative change. Human skills still win on ~25% of tasks (typically highly domain-specific DSLs and conventions). Evolved skills are not just more accurate but more efficient — they compress what the agent must figure out itself.
6. Open Questions: Where Are the Limits?
1. How far can skills evolve? EmbodiSkill's skill bodies stabilize after ~10 revisions — but is that a ceiling of ALFWorld's task space? Unbounded environments might make skills unmanageably complex. 2. Who validates the validator? EmbodiSkill's execution-lapse detection depends on the executor's self-judgment; systematic executor biases could get encoded into the appendix rather than fixed. SKILLEVOLVER's Auditor uses a human-authored checklist — checklist blind spots would accumulate. 3. Cross-domain transfer? Both papers are single-domain (household / code). No evidence yet that ALFWorld-evolved skills transfer to RoboTHOR, or Python-test skills to Rust. 4. Cost-benefit tipping point? At $4/task, a few thousand enterprise tasks cost $12,000+. Neither paper offers a decision framework for when writing skills beats brute-forcing with a big model.
7. Conclusion: Skills as an Interface
Both papers point to a shift: AI is moving from model-centric to skill-centric. Instead of asking "how much stronger is GPT-5 than GPT-4?", we may ask "what skills does this agent have loaded?"
Skills are the interface layer between models and the world — an auditable, reusable, versionable behavioral contract for black-box models. What EmbodiSkill and SKILLEVOLVER demonstrate is this interface layer's capacity for self-evolution: through structured experience extraction and rigorous verification loops, not retraining.
In Feynman's spirit: if you can't explain it simply enough to write into an operating manual, you don't truly understand it. These papers test whether AI can prove understanding by writing operating manuals. In some domains, the AI-written manuals are already better than human experts'.