SkillCraft: When AI Agents Evolve from "Tool Users" to "Senior Architects"
> Paper: SkillCraft: Can LLM Agents Learn to Use Tools Skillfully? > Authors: Shiqi Chen, Jingze Gai, Ruochen Zhou, Jinghan Zhang, Tongyao Zhu, et al. (University of Oxford, City University of Hong Kong, HKUST, Northwestern University, National University of Singapore) > arXiv: https://arxiv.org/abs/2603.00718 > Code: https://github.com/shiqichen17/SkillCraft > Project page: https://skillcraft-website.github.io/page
Why does your AI assistant act like an "amnesiac intern"?
Ask an AI assistant to check the weather, send emails, and manage your schedule daily. It performs well the first week—but each time it reassembles instructions from scratch, repeatedly calling the same APIs as if using them for the first time. On complex tasks (plan a Japan trip: flights, hotels, itinerary, budget), it misses steps, orders things wrongly, or loops on the same tool until it runs out of tokens.
This is the real state of most tool-using agents today: they can call tools, but they cannot use them skillfully.
SkillCraft targets this pain point directly: true intelligence lies not in using scattered tools, but in abstracting complex tool-calling workflows into reusable skills.
The SkillCraft benchmark: testing skill-building, not just tool use
Problems with existing benchmarks
Current tool-use benchmarks (ToolBench, APIBench) mainly measure single-call correctness and task completion. They do not measure:
- Whether an agent can abstract reusable patterns from experience
- Whether it can recall and apply accumulated skills on similar tasks
- Skill quality: elegant composition vs. fragile patchwork
- Agent executes tool calls across tasks
- System identifies frequent, stable tool combinations
- Packages them into parameterized, executable Skills
- Stores them in a persistent skill library
- On new tasks, the agent queries the skill library first
- Matches trigger direct skill invocation; no match falls back to standard tool calling
- New experience is consolidated back into skills
- Skill mode improves efficiency universally: all models reduce token usage with skill reuse
- But skill quality varies enormously: good skills transfer across models; bad skills fail everywhere
- Deep nesting: Skill A calls Skill B calls Skill C—one bug at the bottom collapses everything
- Shallow refinement: each skill wraps only 2–4 atomic tools with well-polished logic, composed via explicit orchestration
- Chen, S., Gai, J., Zhou, R., Zhang, J., Zhu, T., et al. (2026). SkillCraft: Can LLM Agents Learn to Use Tools Skillfully? *arXiv preprint* arXiv:2603.00718.
- Code: https://github.com/shiqichen17/SkillCraft
- Project page: https://skillcraft-website.github.io/page
Dual-dimension difficulty design
SkillCraft contains 126 real API tasks across six domains (coding, finance, entertainment, etc.), with difficulty scaled on two axes:
| Dimension | Description | Example | |------|------|---------| | Quantitative (task volume) | Number of atomic tools involved | 3 → 8 → 15 tools | | Structural (logical complexity) | Dependencies and composition between tools | linear chain → parallel branches → nested conditions → loops |
Four native tools: the agent's "skill workshop"
| Tool | Function | Trigger | |------|------|---------| | Save Skill | Package the current tool-calling chain as reusable code | After completing frequent tasks | | Query Skill | Retrieve a matching skill from the library | When facing a new task | | Read Source | Inspect a skill's implementation | Before reuse | | Execute Skill | Directly invoke a stored skill | Cross-task reuse |
These form a closed loop: complete task → extract pattern → store → retrieve on similar scenarios.
Lightweight evaluation protocol
Phase 1: Skill Acquisition
Phase 2: Skill Reuse
A key design point: no human-written scripts needed. The system traces tool-call trajectories, identifies repeated subsequences, auto-generates parameterized code, and validates by execution before admitting a skill—preventing "garbage skills" from polluting the library.
Results: up to 80% token reduction—but how it drops matters
| Metric | Without skills | With skills | Change | |------|----------|------------|------| | Token usage | baseline | up to 80% lower | significant | | Hard-task success rate | baseline | substantially higher | significant | | Average response time | long (full re-reasoning) | short (direct skill calls) | significant |
Once a "check weather → plan outfit → send reminder" flow is stored as a skill, the agent skips the full reasoning chain next time.
Model comparison: GPT, Claude, DeepSeek
Skill acquisition is not a master key but a mirror: it amplifies differences in each model's underlying induction ability. Models that produce good skills genuinely understand *why* a combination works.
Counterintuitive finding: deep skill nesting causes cascading errors
> Deeply nested skills trigger chained failures; shallow, refined skills offer the best value.
This mirrors a software engineering lesson: composition over inheritance. Good skills are "small, beautiful, composable"—not "big, all-in-one, deeply layered."
What drives success: tool composition ability
> "Success rate strongly correlates with tool composition ability at test time." — from the paper
An agent's success on SkillCraft depends not on single-call accuracy, but on whether it can dynamically compose tools, abstract patterns, and consolidate reuse. This marks an evaluation paradigm shift:
| Old paradigm | New paradigm | |--------|--------| | Single-call accuracy | Cross-task reuse | | Static toolset | Dynamic skill library | | Instance-level success | Skill-level abstraction | | "Can it use tools?" | "Can it build skills?" |
Relation to prior work
| Work | Core capability | Difference from SkillCraft | |------|---------|------| | ToolLLM | Large-scale tool learning | No skill abstraction/reuse measurement | | LATM / CREATOR | AI creating its own tools | Creates new tools rather than composing existing ones | | Voyager | Experience to skill library | Minecraft environment, not real APIs | | MemSkill | Memory-skill evolution | Focus on memory, not skill quality evaluation | | SkillCraft | Automatic skill acquisition + cross-task reuse + quality evaluation | First dual-dimension difficulty + lightweight protocol |
Limitations and future directions
Limitations: 1. Skill granularity is set by predefined system rules and may be suboptimal 2. 126 tasks across six domains cover only a slice of the real API ecosystem 3. Skill conflicts (multiple matching skills) are unresolved 4. Long-horizon degradation: does shallow skill composition stay efficient over very long chains?
Future directions: 1. Adaptive granularity: let agents decide whether a pattern is worth saving 2. Skill versioning: versioning, updates, rollback 3. Cross-agent skill sharing 4. Skill marketplaces: a decentralized skill ecosystem
Conclusion: from "tool user" to "architect"
SkillCraft is more than a benchmark—it's a paradigm statement:
> The next stage of AI tool use is not "more tools" but "better skills."
Current agent frameworks focus on enabling more API calls (MCP, Function Calling, tool retrieval)—solving "can it use tools?" SkillCraft addresses "does it use them well?": escaping repetitive labor, consolidating experience into reusable assets, and calling accumulated tool chains the way a senior engineer does. When an agent develops "muscle memory"—invoking verified skill combinations instead of reasoning from scratch every time—it evolves from a tool user into a senior architect.