English static mirror for SEO/GEO · AI-assisted translation · Read Chinese original

Anatomy of the Agent Skill Lifecycle: 75% Help, 25% Hurt

Forum topic · 小凯 · 2026-05-25

Summary

A forum post on zhichai.net reviews a systematic study (arXiv:2605.23899) from Fudan, Zhejiang University, Microsoft and other institutions that dissects the full lifecycle of model-generated Agent Skills: experience generation, skill extraction, and skill consumption. Using 5 task domains (ALFWorld, SpreadsheetBench, SWE-bench-Verified, SEAL-0, BFCL-v4), 6 target models and 5 extractor models, the study builds a 150-datapoint evaluation matrix. Key findings: 75% of skill deployments show positive transfer but 25% cause negative transfer (embodied ALFWorld is most fragile at 47%); a model's task-solving ability does not predict its skill-extraction quality (GPT-5.4 ranks last as extractor while Gemini-3.1-Flash-Lite ranks best); skill utility is driven by content, not text format; human/LLM judgments of skill text correlate only 0.31 with actual downstream performance; and optimal success/failure trajectory ratios are domain-specific. The paper proposes a zero-training utility-aware meta-skill extraction framework with negative-transfer suppression, and argues skills need online validation loops rather than static reuse.

Overview

Agent Skills have become standard for LLM agents (Anthropic's Agent Skills protocol, Alibaba's Trace2Skill, SkillRL), but one basic question remains unanswered: across the full chain from experience to skill to consumption, what actually determines a skill's downstream utility? Existing work each covers only part of the lifecycle: SkillsBench (human-written seed skills), SWE-Skills-Bench (public skill libraries), Trace2Skill (extraction stage), SkillRL (skill-augmented RL). A new paper from Fudan, Zhejiang University, Microsoft and others fills this gap with a unified framework spanning 5 domains, 6 target models, 5 extractor models — a 150-datapoint experimental matrix.

Framework design

The lifecycle is formalized as three stages:

1. Experience generation: target model M executes tasks on domain D's training set, producing a pool T_M,D = {(task_i, trajectory_i, outcome_i)}. 2. Skill extraction: extractor model E distills the pool into a skill set S = E(T_M,D). 3. Skill consumption: the same target model M uses skills S on the test set; performance change Δ(E,M,D) = Perf(M|S) − Perf(M|∅).

Domains covered: embodied interaction (ALFWorld), productivity software (SpreadsheetBench), software engineering (SWE-bench-Verified), web search (SEAL-0), tool calling (BFCL-v4). Models: GPT-5.4, GPT-5.4-mini, Gemini-3.1-Pro, Gemini-3.1-Flash-Lite, Qwen3.5-35B (as target + extractor), plus Qwen3.5-9B (target only).

Key findings

  • 75% positive transfer, 25% negative transfer. Generated skills help on average but are not universally beneficial. ALFWorld is the most fragile domain (47% negative transfer) — formalized skill constraints can limit exploration in physically-grounded, multi-step tasks. SpreadsheetBench and SWE-bench-Verified are most stable (87% positive).
  • Extraction ability ≠ consumption ability. GPT-5.4 has the strongest baseline on SpreadsheetBench but ranks last as an extractor (+1.67pp effectiveness), while Gemini-3.1-Flash-Lite extracts best (+5.86pp). Being good at a task does not mean being good at summarizing reusable experience from it — optimal architectures may split the roles.
  • Skill utility is target-model-specific. Skills extracted by GPT-5.4 cause negative transfer on Gemini-3.1-Pro, Gemini-3.1-Flash-Lite, and Qwen3.5-35B on ALFWorld. A skill that helps one model can be poison for another.
  • Failure trajectories matter — with domain-specific optima. Testing success ratios of 0–100% in the experience pool: pure-failure pools are always worst, but ALFWorld peaks at 25–50% success (failure-heavy is better), while SpreadsheetBench and SWE-bench-Verified peak at 75–100%. Success trajectories give positive procedural signals; failures give negative constraint signals that shrink the search space in exploratory domains.
  • Content drives utility, not format. Switching skill text formats (ordered/unordered lists, checklists, prose) shows no significant effect (Friedman p>0.34), while changing extractor is significant on 5/6 targets (p<<0.01). Focus on content accuracy and coverage, not formatting.
  • Text plausibility decouples from real utility. GPT-5.4 judging skill text picks the "better" skill with only 0.31 correlation to downstream performance. Skill marketplaces need dynamic, consumption-based quality ratings, not expert review alone.
  • Meta-skill framework

    The paper proposes a utility-oriented meta-skill extraction strategy:

  • Utility-aware extraction: the extractor predicts each candidate skill's expected downstream utility and prioritizes high-utility skills.
  • Negative-transfer suppression: skills causing negative transfer are automatically rolled back and flagged high-risk, with reduced weight in later extractions.
  • Cross-domain meta-skills: patterns valid across domains ("verify input format before processing", "fall back to defaults on anomalies") are more stable than domain-specific skills.
  • The framework improves skill quality across domains with no additional training — implemented purely via prompt engineering, so it can be integrated into existing extraction pipelines at zero cost.

    Strategic implications

  • Skill utility is fundamentally not fully predictable under current technology: extractor–target interactions are nonlinear, text quality decouples from utility, and optimal experience ratios are domain-specific. Systems need an online validation loop: generate → small-scale A/B test → keep/discard → iterate. Static "extract once, use forever" does not work.
  • Complementary to Trace2Skill (arXiv:2603.25158): Trace2Skill showed extraction can work spectacularly (+57.65pp for Qwen3.5-122B on WikiTableQuestions with Qwen3.5-35B-extracted skills), but that success relied on specific task-model combinations. This paper explains when things go wrong.
  • Skills should not be managed as static assets. Ecosystems may need dynamic adaptation layers, utility-tracking systems, and A/B-testing infrastructure for new skills.
  • Conclusion

    The paper's core value is dismantling myths with systematic evidence:

  • "Stronger models extract better skills" — false.
  • "More professional-looking skill text works better" — false (0.31 correlation).
  • "More success experience is always better" — false (ALFWorld's optimum is failure-heavy).
  • "Skills can be extracted once and reused anywhere" — false (cross-model transfer is highly uncertain).
  • The pragmatic contribution is the zero-training meta-skill framework. The future of skill automation is not "AI writes perfect skills," but "a verifiable, rollback-capable, adaptive skill production pipeline." This paper lays the first foundation stone.

    References

  • Paper: From Raw Experience to Skill Consumption: A Systematic Study of Model-Generated Agent Skills (arXiv:2605.23899)
  • Trace2Skill: Distill Trajectory-Local Lessons into Transferable Agent Skills (arXiv:2603.25158)
  • Anthropic Agent Skills protocol; SkillRL; open Agent Skills standards

Tags

#agent-skills#llm-agents#skill-extraction#negative-transfer#meta-skills#ai-research#benchmarking#skill-automation

This page is an English static mirror generated for search and AI citation. It may be a full translation or structured summary of the Chinese original. Canonical interactive discussion lives on the Chinese page: https://zhichai.net/topic/177620804