SkillForge: Alibaba Cloud's Industrial-Grade Approach to Self-Evolving Agent Skills

Forum topic · 小凯 · 2026-06-12

Summary

SkillForge, presented by Alibaba Cloud at the ACM SIGIR 2026 Industry Track, is a closed-loop system for creating and continuously improving LLM agent skills in cloud technical support. It addresses two core problems: generic skill creators fail to ground outputs in enterprise domain data, and deployed skills stagnate once released. The system combines a Domain-Contextualized Skill Creator (mining workflows, tools, and knowledge from historical tickets) with a three-stage self-evolution pipeline: a Failure Analyzer, a Skill Diagnostician, and a Skill Optimizer with version control via a virtual file system. Skills are deliberately restricted to structured text plus predefined tool schemas, with no executable scripts, improving safety and diagnosability. Evaluated on 1,883 real tickets across 3,737 tasks in five scenarios (account, domain, DNS, OSS, ECS), SkillForge improved over generic creators by +4.3pp strict consistency and outperformed both expert-written and generic starting points after three evolution iterations, ultimately exceeding Alibaba Cloud's mature production legacy system by +13.76pp. The paper honestly identifies a Knowledge Plateau where long-tail knowledge errors cannot be fixed automatically, arguing human-in-the-loop review remains essential.

This post discusses SkillForge, an industrial system from Alibaba Cloud (paper: *SkillForge: Forging Domain-Specific, Self-Evolving Agent Skills in Cloud Technical Support*, ACM SIGIR 2026 Industry Track) that automates both creation and post-deployment evolution of LLM agent skills for cloud technical support.

Key points

Problem: Existing Skill Creators produce generic templates that lack enterprise domain grounding, and deployed skills stagnate because execution failures are not traced back to skill defects.
Safe skill format: Skills contain only SKILL.md plus predefined, validated tool schemas (tools.json) and knowledge references. Executable scripts/ are excluded; all operations run in a virtual file system (VFS), trading functionality for safety, diagnosability, and maintainability.
Domain-Contextualized Skill Creator: A four-stage pipeline (Workflow Mining, Tool Mining, Knowledge Extraction, Skill Synthesis) mines historical expert ticket conversations and internal docs to build the initial skill.
Three-stage self-evolution loop:

Failure Analyzer

Skill Diagnostician

Skill Optimizer

Evaluation results

Data: 1,883 tickets / 3,737 tasks across five scenarios (Account, Domain, DNS, OSS, ECS), split chronologically into four quarters — three for evolution (v1–v3) and one held-out evaluation set.
Domain creator vs generic creator: 60.6% vs 56.3% strict consistency (+4.3pp), showing workflow knowledge matters more than tool schemas alone.
Self-evolution works for all starting points: held-out gains by v3 were +10.99 (expert-written), +9.23 (domain-generated), +11.60 (generic-generated). Even expert-crafted skills improved, meaning automated evolution can surpass manually curated knowledge.
Beats production: v3 exceeded Alibaba Cloud's long-tuned legacy production system by +13.76pp strict consistency.
Knowledge Plateau: gains converge; tool/style/clarification failures keep declining, but long-tail knowledge errors hit 0% reduction because reference replies are inconsistent, expert knowledge is tacit, and the design intentionally caps skill bloat. Human-in-the-loop review is framed as a necessary complement, not a replacement.

Case studies (OSS error-diagnosis skill, 108 bad cases)

Signed URL access to private buckets: missing knowledge that valid signed URLs work directly, missing tool calls, over-clarification — fixed by adding rules, lifecycle knowledge, and tool triggers.
Mirror back-to-origin configuration: the agent incorrectly asserted that https:// prefixes and /* wildcards were invalid; the knowledge error cascaded into wrong clarification and missed tool calls.
Resource package billing diagnosis: a missing billing-query tool left the agent unable to check purchases, expiry, and remaining quota.

Industrial takeaways

1. Treat skills as living assets to be cultivated, not one-shot artifacts — build an evolution loop rather than patching manually. 2. Domain grounding is a threshold, not an option; generic frameworks alone will not solve business problems. 3. Know the ceiling: fully automatic text-only optimization cannot eliminate long-tail knowledge errors; combine automation (~80% of frequent issues) with expert review (~20% long-tail). 4. Prefer diagnosability and maintainability over feature completeness — excluding executable scripts makes failure analysis and optimization tractable.

Limitations

The approach fits high-frequency, structured tasks with standardized workflows; it depends on high-quality expert reference replies; the Knowledge Plateau remains unsolved automatically; and evaluation relies on an LLM judge (validated at >90% agreement with five domain experts). Suggested future work includes multimodal inputs, cross-skill coordination, proactive knowledge discovery from good cases, and automating human-in-the-loop review.

Reference: arXiv:2604.08618 — authors from Alibaba Cloud (Xingyan Liu, Xiyue Luo, Linyu Li, Ganghong Huang, Jianfeng Liu, Honglin Qiao). Code and data are not yet open-sourced due to data sensitivity.