SkillForge: Alibaba Cloud's Industrial Practice in Self-Evolving Agent Skills

Forum topic · 小凯 · 2026-06-12

Summary

Alibaba Cloud's SkillForge addresses two core problems in enterprise LLM agent skills: generic skill creators lack domain grounding, and skills stagnate after deployment. Built on 1,883 real cloud-support tickets across five scenarios (Account, Domain, DNS, OSS, ECS; 3,737 tasks), SkillForge combines a domain-contextualized skill creator with a three-stage self-evolution loop: a Failure Analyzer that examines knowledge, tool, clarification, and style dimensions; a Skill Diagnostician using ReAct-style root-cause attribution; and a Skill Optimizer applying minimal, evidence-based edits with VFS versioning. For security and maintainability, skills contain no executable scripts, only predefined tool schemas and markdown knowledge. Results show domain-grounded creation beats generic creation (+4.3pp strict consistency) and the evolution loop improves every starting point, including expert-written skills (up to +11.60pp), eventually outperforming Alibaba Cloud's legacy production system by +13.76pp. The paper honestly identifies a 'Knowledge Plateau' where long-tail knowledge errors resist fully automated optimization, arguing human-in-the-loop review remains essential.

This post analyzes SkillForge: Forging Domain-Specific, Self-Evolving Agent Skills in Cloud Technical Support (Alibaba Cloud, ACM SIGIR 2026 Industry Track, arXiv:2604.08618).

Key points

Problem: Existing skill creators produce generic templates not grounded in enterprise data and tools; after deployment, skills stagnate because execution failures aren't traced back to skill defects.
Two goals: (1) domain-grounded initial skill creation, (2) systematic post-deployment self-evolution.
Security-first skill format: skills contain only SKILL.md (structured instructions) plus references/ (validated tool schemas, knowledge docs). No scripts/ directory; all operations via a virtual file system (VFS) with no arbitrary code execution.

Architecture: a closed loop

1. Domain-Contextualized Skill Creator: four stages — workflow mining (extracting expert solution patterns from historical ticket conversations), tool mining (building tools.json), knowledge extraction (from internal docs), and skill synthesis into a fixed five-section template (Background Knowledge, Case-Type Triage, Per-Case-Type Handling, FAQ, Reference Index). 2. Execution & monitoring on live traffic. 3. Failure Analyzer: parallel four-dimensional analysis (Knowledge / Tool / Clarification / Style), aggregated into fail / marginal / acceptable verdicts, prioritized knowledge > tool > clarification > style. 4. Skill Diagnostician: batch-aggregates 100+ bad cases, then a ReAct-style agent maps failures to specific SKILL.md locations (missing / insufficient / incorrect) and generates a machine-parseable optimization plan. 5. Skill Optimizer: targeted edits under three principles — minimal modification, do no harm, evidence-based — plus a three-tier knowledge placement strategy (frequent stable knowledge in SKILL.md, long-tail in FAQ, well-documented topics as references). Each revision commits a versioned VFS state.

Experimental results (1,883 tickets, 3,737 tasks)

Five scenarios: Account (389), Domain (527), DNS (256), OSS (385), ECS (326); data split chronologically into four quarters with the last as held-out evaluation. LLM-judge agreement with 5 domain experts: >90%.
Domain vs. generic creator: 60.6% vs. 56.3% strict consistency (+4.3pp; largest gain +7.2pp on ECS) — workflow knowledge matters beyond tool schemas.
Evolution loop gains on held-out data (strict CR, by v3): expert-written +10.99, domain-generated +9.23, generic-generated +11.60 — universal, monotonically increasing improvement, strongest for weak starting points, and automated evolution surpasses manually curated skills.
vs. legacy production system: +13.76pp strict CR for v3.

Limitations: the Knowledge Plateau

Marginal gains shrink over iterations (e.g., S_manual: +4.09 → +5.55 → +1.35). Tool, style, and clarification failures keep declining, but knowledge-related improvements hit zero by v3 due to: reference inconsistency (experts contradict each other), deliberate skill-bloat constraints, and tacit expert knowledge unrecoverable from text. Fully automated text-only optimization has an inherent ceiling — human-in-the-loop review is necessary for long-tail errors.

Industrial takeaways

1. Don't write skills from scratch — mine them from historical data. 2. Don't stop at deployment — build an evolution loop. 3. Don't chase full automation — insert human review where needed. 4. Don't prioritize functional completeness — prioritize diagnosability and maintainability (excluding executable scripts makes failure analysis and optimization tractable).

Other limitations: the method assumes high-frequency structured tasks and high-quality reference replies; evaluation relies on LLM-judges with possible systematic bias. Future directions include multimodal inputs, cross-skill coordination, learning from good cases, and automating the human review step.

Reference: arXiv:2604.08618 — Xingyan Liu, Xiyue Luo, Linyu Li, Ganghong Huang, Jianfeng Liu, Honglin Qiao (Alibaba Cloud), ACM SIGIR 2026 Industry Track. Code and data not yet released.