Lifecycle Anatomy of Model-Generated Agent Skills: 75% Effective, 25% Negative Transfer

Forum topic · 小凯 · 2026-05-25

Summary

This systematic study by Fudan, Zhejiang University, and Microsoft researchers dissects the full lifecycle of model-generated agent skills—experience generation, skill extraction, and skill consumption—using a unified evaluation framework spanning 5 task domains (ALFWorld, SpreadsheetBench, SWE-bench-Verified, SEAL-0, BFCL-v4), 6 target models, and 5 extractors, producing 150 data points. Key findings: 75% of configurations show positive transfer (Δ>0) while 25% show negative transfer; ALFWorld is most fragile (47% negative), SpreadsheetBench and SWE-bench most stable (87% positive). Counter-intuitively, stronger task-solving models are not better skill extractors—GPT-5.4 ranks last as an extractor despite top baseline performance. Optimal success/failure trajectory ratios are domain-specific: exploration-heavy domains benefit from failure-heavy pools, while rule-based domains prefer success-heavy pools. Text format (lists, checklists, prose) has no significant effect; content drives variance. Static text quality correlates only 0.31 with downstream utility. The paper proposes a zero-training meta-skill framework with utility-aware extraction and negative-transfer suppression.

Lifecycle Anatomy of Model-Generated Agent Skills

1. The Problem: Blind Spots in Skill Automation

By 2026, Agent Skills have become standard equipment for LLM-based agents. Anthropic's Agent Skills protocol, Alibaba's Trace2Skill framework, and SkillRL's recursive skill evolution represent a flourishing landscape—but a foundational question remains unanswered: what truly determines the downstream utility of skills across the full pipeline from experience to consumption?

Existing work covers fragments of the lifecycle: SkillsBench uses hand-written seed skills for benchmarking; SWE-Skills-Bench draws from public skill libraries; Trace2Skill focuses on extraction; SkillRL studies RL-based skill augmentation. No one has systematically investigated the complete chain of experience generation → skill extraction → skill consumption.

This paper from Fudan, Zhejiang University, and Microsoft fills that gap with a unified evaluation framework spanning 5 task domains, 6 target models, and 5 extractor models—producing a complete 150-data-point experimental matrix.

2. Experimental Design: A Full-Lifecycle Evaluation Framework

#### 2.1 Three-Stage Formal Definition

Stage 1 – Experience Generation: Target model M executes tasks on domain D's training set, producing experience pool T_{M,D} = {(task_i, trajectory_i, outcome_i)}.
Stage 2 – Skill Extraction: Extractor model E distills T_{M,D} into skill set S_{E,M,D} = E(T_{M,D}).
Stage 3 – Skill Consumption: Same target model M uses skills S on test set; measure Δ(E,M,D) = Perf(M|S) − Perf(M|∅).

#### 2.2 Five Task Domains

| Domain | Benchmark | Core Capability | |--------|-----------|-----------------| | Embodied Interaction | ALFWorld | Physical common sense, exploration, multi-step planning | | Productivity Software | SpreadsheetBench | Spreadsheet inspection, formula reasoning, value editing | | Software Engineering | SWE-bench-Verified | Codebase understanding, bug localization, patch generation | | Web Search | SEAL-0 | Retrieval, evidence synthesis, multi-hop reasoning | | Tool Calling | BFCL-v4 | Function selection, parameter extraction, multi-turn tool use |

#### 2.3 Model Matrix: 6 Targets × 5 Extractors

Models include GPT-5.4, GPT-5.4-mini, Gemini-3.1-Pro, Gemini-3.1-Flash-Lite, Qwen3.5-35B (target+extractor), and Qwen3.5-9B (target only).

3. Core Findings: 75% Effective, 25% Negative Transfer

#### 3.1 Overall Picture

The complete Δ matrix reveals a complex landscape: 75% of entries show positive transfer (Δ>0), but 25% show negative transfer (Δ<0). Model-generated skills help on average, but are not universally beneficial.

| Domain | Positive Transfer | Negative Transfer | Fragility | |--------|-------------------|-------------------|-----------| | ALFWorld | 53% | 47% | Most fragile | | SpreadsheetBench | 87% | 13% | Most stable | | SWE-bench-Verified | 87% | 13% | Most stable | | SEAL-0 | 70% | 30% | Moderate | | BFCL-v4 | 70% | 30% | Moderate |

ALFWorld's 47% negative transfer rate is striking—physical common-sense tasks with multi-step exploration suffer when formal skill constraints limit the agent's exploration space.

#### 3.2 Counter-Intuitive: Extraction Capability ≠ Consumption Capability

The most counter-intuitive finding: a model's task-solving ability does not predict its skill extraction quality.

GPT-5.4 has the strongest SpreadsheetBench baseline (37.17%), but as an extractor its Extraction Efficacy (EE) ranks last (+1.67pp). Conversely, Gemini-3.1-Flash-Lite achieves the highest EE (+5.86pp) as an extractor, despite not having the strongest baseline.

Optimal configuration: let a strong model execute tasks to produce trajectories, and a separate model specialize in extraction—rather than having the same model both execute and summarize.

#### 3.3 Sharp Differences in Target Evolvability

The same extractor produces wildly different effects across target models. On ALFWorld, GPT-5.4-extracted skills yield +4.23pp on GPT-5.4, +2.84pp on GPT-5.4-mini, but −0.15pp on Gemini-3.1-Pro, −1.59pp on Gemini-3.1-Flash-Lite, and −1.34pp on Qwen3.5-35B.

Skill consumption is model-specific—a skill extracted by one model can be poison for another.

4. Lifecycle Deep Dive: What Drives Skill Utility

#### 4.1 Experience Generation: The Value of Failed Trajectories

Testing 5 success ratios (100%, 75%, 50%, 25%, 0%) with fixed extractor GPT-5.4-mini:

Pure failure pool always performs worst
Optimal ratio is domain-specific
ALFWorld prefers 25%–50% success (failure-heavy)
SpreadsheetBench and SWE-bench-Verified prefer 75%–100% success

Two experience types play distinct roles: successful trajectories provide positive procedural signals; failed trajectories provide negative constraint signals.

#### 4.2 Skill Extraction: Content-Driven, Not Form-Driven

Testing skill text formats (ordered lists, unordered lists, checklists, prose): Friedman test p>0.34, σ-ratio<1—format effects do not exceed run noise.

Switching extractors produces significant effects on 5/6 targets (p<<0.01, σ-ratio>1). Variance is driven by skill content, not form.

#### 4.3 Skill Consumption: Disconnect Between Text Credibility and Utility

GPT-5.4 as judge, selecting "better skills" based on text alone, correlates only 0.31 with downstream task performance.

If skill quality cannot be predicted via static text evaluation, future Skills Stores will need dynamic, consumption-based quality rating—not expert review alone.

5. Meta-Skill Framework: From Problem to Solution

The paper proposes a utility-oriented meta-skill extraction strategy:

Utility-aware extraction: Extractor predicts each candidate skill's expected downstream utility and prioritizes high-utility candidates.
Negative-transfer suppression: Auto-rollback and flag high-risk skills when consumption produces negative transfer.
Cross-domain meta-skills: Identify patterns effective across multiple domains (e.g., "validate input format before processing," "fall back to default on exception").

Experiments show stable skill quality improvement and significantly reduced negative transfer. The framework requires no additional training—fully prompt-engineering-based and integrable at zero cost.

6. Strategic Review: The Real Boundaries of Skill Automation

#### 6.1 Why Skill Utility Is Fundamentally Unpredictable

Extractor-target interaction is nonlinear
Skill text quality is decoupled from actual utility
Domain characteristics determine optimal experience ratios

Skill automation systems need online verification loops: generate → small-scale A/B test → retain effective, discard negative-transfer → iterate extraction.

#### 6.2 Complementarity with Trace2Skill

Alibaba's Trace2Skill (arXiv:2603.25158) showed Qwen3.5-35B-extracted skills boosting Qwen3.5-122B by 57.65pp on WikiTableQuestions. This paper reveals that such successes are not universal—cross-domain, cross-model-family transfer is highly uncertain. Together, the two papers provide a complete picture: Trace2Skill shows "it can work very well," this paper shows "when it won't."

#### 6.3 Implications for the Agent Ecosystem

Skills should not be managed as static assets. Current Anthropic Agent Skills protocols and Skills Stores treat skills as installable, reusable static modules—but skill consumption effects are highly context-dependent. The same skill may produce +14pp on one model-task combination and −3pp on another.

Future ecosystems will need: dynamic adaptation layers, utility tracking systems, and A/B testing infrastructure.

7. Conclusion

The paper's greatest value lies in systematically dismantling several myths about skill automation:

*"Stronger models extract better skills"* — False. GPT-5.4 task-strongest, extractor-last.
*"More professionally written skills work better"* — False. Text quality correlates only 0.31 with utility.
*"More successful experience is always better"* — False. ALFWorld's optimum is failure-heavy.
*"Skills can be extracted once and reused everywhere"* — False. Cross-model transfer is highly uncertain.

The practical contribution is the meta-skill framework—a zero-training-cost extraction strategy improvement, immediately deployable. The future of skill automation is not "AI automatically writes perfect skills," but "verifiable, rollback-capable, adaptable skill production pipelines."

---

References & Further Reading

Paper: *From Raw Experience to Skill Consumption: A Systematic Study of Model-Generated Agent Skills* (arXiv:2605.23899)
Trace2Skill: *Distill Trajectory-Local Lessons into Transferable Agent Skills* (arXiv:2603.25158)
Anthropic Agent Skills Protocol
SkillRL: *Recursive Skill-Augmented RL for Agent Evolution*
Agent Skills Open Standard