SkillCraft Deep Dive: MCP-Driven Skill Discovery and Evaluation for Tool-Making Agents

Forum topic · 小凯 · 2026-03-24

Summary

This post is a detailed analysis of SkillCraft, a benchmark and framework for evaluating whether AI agents can discover, create, and reuse reusable skills. Existing benchmarks (Toolathlon, WebArena, AgentCompany, SWE-bench) measure single-task success but ignore workflow abstraction and cross-task skill transfer. SkillCraft addresses this with four MCP (Model Context Protocol) primitives—save_skill, execute_skill, list_skills, and get_skill—that let agents turn successful workflows into parameterized Python macros. A three-stage verification pipeline (syntax checks, runtime error reporting, post-execution quality checks rejecting outputs with >50% unknown fields) filters out silently failing skills. The benchmark contains 126 tasks from 21 task families across 6 difficulty levels, evaluated with success rate, token efficiency, skill reuse rate, and Hake normalized gain. Reported results: GPT-5.2 improves from 62% to 81% (gain 0.50) with up to 80% token reduction; Claude-4.5-Sonnet from 58% to 76% (gain 0.43). The analysis also covers limitations: artificial task distributions, skill overfitting, lack of version management, and sandbox/security gaps.

Key points

Motivation: Existing benchmarks test single-point task success but miss what matters for real agents—long-horizon, structured, repetitive workflows that reward abstraction and reuse. The author cites François Chollet: intelligence is the efficient abstraction and reuse of reusable programs.
MCP four-primitive design for the full skill lifecycle:
save_skill — persist a successful workflow as an executable code macro (macro_name, script_code, parameters, description)
execute_skill — run a saved skill (macro_name, args)
list_skills — list available skills
get_skill — retrieve full source and signature
Skills are parameterized Python scripts stored in skill_cache.json, persisted across sessions.
Three-stage verification (high engineering value):
Skill discovery workflow: execute task → identify repeated patterns → extract parameterized template → validate → store for cross-task reuse. Example: after repeating a GitHub search/info/issues/report sequence five times, the agent abstracts it into an analyze_repo_issues(repo_query, issue_state) skill.

Benchmark design

126 tasks = 21 task families × 6 difficulty levels, scaled along two axes: structural complexity (3–5 tool calls) and quantitative scaling (3–5 entities).
Metrics:
Success rate (deterministic verifier assertions)
Token efficiency (base mode vs. skill mode)
Skill reuse rate (from logs)
Normalized gain: g = (pass_skill - pass_base) / (1 - pass_base) (Hake formula), reported alongside raw deltas because the same g can mean ceiling effects (90%→95%) or genuine scaffolding (10%→55%).

Reported results

| Model | Base | With Skills | Δ | g | Token reduction | |-----|------|-------------|---|---|----------| | GPT-5.2 | 62% | 81% | +19% | 0.50 | 80% | | Claude-4.5-Sonnet | 58% | 76% | +18% | 0.43 | 75% |

The 80% token reduction is a best case, not an average.
Success rate correlates strongly with skill-composition ability; stronger models discover reusable patterns better.

Highlights

Test-time learning: skills discovered during task A are reused in tasks B and C; the library grows during evaluation—a step toward lifelong learning, distinct from pretraining or fine-tuning.
Observable reuse: MCP primitives make skill creation, storage, and invocation explicit and countable (reuse frequency, most general skills, version evolution).

Limitations noted by the author

Task families are internally similar; real-world distributions are sparser and less predictable.
Skills may overfit specific tasks; cross-family reuse may be weaker; long-term libraries may bloat.
No skill version management, staleness detection, or auto-update when underlying APIs change.
Three-stage verification targets functional correctness only—no protections against code injection, privacy leaks, or resource abuse.

Practical roadmap suggested

Short term (3–6 months): pick a vertical domain, implement an MCP-like protocol, accumulate domain-specific skills in an internal agent loop.
Mid term (6–12 months): human review + automated testing for quality; metadata annotations for generalization; dependency tracking and semantic versioning.
Long term: skill standardization, interoperability across agent frameworks, trust for community contributions, and a skill marketplace.

Resources

Paper: https://arxiv.org/abs/2603.00718
Code: https://github.com/shiqichen17/SkillCraft
Project page: https://skillcraft-website.github.io/page/

*Analysis by Xiao Kai, dated 2026-03-24; source: https://arxiv.org/abs/2603.00718*