Overview
Agent Skills have become standard for LLM agents (Anthropic's Agent Skills protocol, Alibaba's Trace2Skill, SkillRL), but one basic question remains unanswered: across the full chain from experience to skill to consumption, what actually determines a skill's downstream utility? Existing work each covers only part of the lifecycle: SkillsBench (human-written seed skills), SWE-Skills-Bench (public skill libraries), Trace2Skill (extraction stage), SkillRL (skill-augmented RL). A new paper from Fudan, Zhejiang University, Microsoft and others fills this gap with a unified framework spanning 5 domains, 6 target models, 5 extractor models — a 150-datapoint experimental matrix.
Framework design
The lifecycle is formalized as three stages:
1. Experience generation: target model M executes tasks on domain D's training set, producing a pool T_M,D = {(task_i, trajectory_i, outcome_i)}. 2. Skill extraction: extractor model E distills the pool into a skill set S = E(T_M,D). 3. Skill consumption: the same target model M uses skills S on the test set; performance change Δ(E,M,D) = Perf(M|S) − Perf(M|∅).
Domains covered: embodied interaction (ALFWorld), productivity software (SpreadsheetBench), software engineering (SWE-bench-Verified), web search (SEAL-0), tool calling (BFCL-v4). Models: GPT-5.4, GPT-5.4-mini, Gemini-3.1-Pro, Gemini-3.1-Flash-Lite, Qwen3.5-35B (as target + extractor), plus Qwen3.5-9B (target only).
Key findings
- 75% positive transfer, 25% negative transfer. Generated skills help on average but are not universally beneficial. ALFWorld is the most fragile domain (47% negative transfer) — formalized skill constraints can limit exploration in physically-grounded, multi-step tasks. SpreadsheetBench and SWE-bench-Verified are most stable (87% positive).
- Extraction ability ≠ consumption ability. GPT-5.4 has the strongest baseline on SpreadsheetBench but ranks last as an extractor (+1.67pp effectiveness), while Gemini-3.1-Flash-Lite extracts best (+5.86pp). Being good at a task does not mean being good at summarizing reusable experience from it — optimal architectures may split the roles.
- Skill utility is target-model-specific. Skills extracted by GPT-5.4 cause negative transfer on Gemini-3.1-Pro, Gemini-3.1-Flash-Lite, and Qwen3.5-35B on ALFWorld. A skill that helps one model can be poison for another.
- Failure trajectories matter — with domain-specific optima. Testing success ratios of 0–100% in the experience pool: pure-failure pools are always worst, but ALFWorld peaks at 25–50% success (failure-heavy is better), while SpreadsheetBench and SWE-bench-Verified peak at 75–100%. Success trajectories give positive procedural signals; failures give negative constraint signals that shrink the search space in exploratory domains.
- Content drives utility, not format. Switching skill text formats (ordered/unordered lists, checklists, prose) shows no significant effect (Friedman p>0.34), while changing extractor is significant on 5/6 targets (p<<0.01). Focus on content accuracy and coverage, not formatting.
- Text plausibility decouples from real utility. GPT-5.4 judging skill text picks the "better" skill with only 0.31 correlation to downstream performance. Skill marketplaces need dynamic, consumption-based quality ratings, not expert review alone.
- Utility-aware extraction: the extractor predicts each candidate skill's expected downstream utility and prioritizes high-utility skills.
- Negative-transfer suppression: skills causing negative transfer are automatically rolled back and flagged high-risk, with reduced weight in later extractions.
- Cross-domain meta-skills: patterns valid across domains ("verify input format before processing", "fall back to defaults on anomalies") are more stable than domain-specific skills.
- Skill utility is fundamentally not fully predictable under current technology: extractor–target interactions are nonlinear, text quality decouples from utility, and optimal experience ratios are domain-specific. Systems need an online validation loop: generate → small-scale A/B test → keep/discard → iterate. Static "extract once, use forever" does not work.
- Complementary to Trace2Skill (arXiv:2603.25158): Trace2Skill showed extraction can work spectacularly (+57.65pp for Qwen3.5-122B on WikiTableQuestions with Qwen3.5-35B-extracted skills), but that success relied on specific task-model combinations. This paper explains when things go wrong.
- Skills should not be managed as static assets. Ecosystems may need dynamic adaptation layers, utility-tracking systems, and A/B-testing infrastructure for new skills.
- "Stronger models extract better skills" — false.
- "More professional-looking skill text works better" — false (0.31 correlation).
- "More success experience is always better" — false (ALFWorld's optimum is failure-heavy).
- "Skills can be extracted once and reused anywhere" — false (cross-model transfer is highly uncertain).
- Paper: From Raw Experience to Skill Consumption: A Systematic Study of Model-Generated Agent Skills (arXiv:2605.23899)
- Trace2Skill: Distill Trajectory-Local Lessons into Transferable Agent Skills (arXiv:2603.25158)
- Anthropic Agent Skills protocol; SkillRL; open Agent Skills standards
Meta-skill framework
The paper proposes a utility-oriented meta-skill extraction strategy:
The framework improves skill quality across domains with no additional training — implemented purely via prompt engineering, so it can be integrated into existing extraction pipelines at zero cost.
Strategic implications
Conclusion
The paper's core value is dismantling myths with systematic evidence:
The pragmatic contribution is the zero-training meta-skill framework. The future of skill automation is not "AI writes perfect skills," but "a verifiable, rollback-capable, adaptive skill production pipeline." This paper lays the first foundation stone.