Trace2Skill Explained: Distilling Agent Trial-and-Error into Transferable Skills

> One-sentence takeaway: Rather than having an agent memorize the details of every failure (retrieval-style experience banks), Trace2Skill distills a few hundred trial-and-error runs into a handful of general-purpose "craft rules"—like a master craftsman teaching an apprentice not "you stripped the third screw last time," but "this kind of screw needs three hand turns first." The paper shows skills distilled by a 35B model can lift a 122B model by 57 percentage points.

1. A Concrete Scenario

Imagine teaching an intern to process Excel spreadsheets. First attempt: they write formulas but forget to recalculate, so cells display empty. You correct them: "run recalc.py after writing formulas." Second attempt: they save with pandas, silently flattening formulas into static values. Again: "use openpyxl for edits—pandas destroys formula relationships." Third attempt: deleting rows top-down scrambles indices...

If you only ever say "here's where you went wrong last time," the intern never learns a systematic method. But if you consolidate these lessons plus rules like "re-open the file to verify after writing" and "back up before editing" into an Excel handling standard—that's a skill.

Trace2Skill automates exactly this "from trial-and-error to specification" distillation.

2. The Core Idea: Extract Patterns, Don't Memorize Errors

The paper's key insight: the value of experience lies not in remembering the specifics of each failure, but in discovering general patterns behind failures.

Problems with existing approaches:

• Retrieval-style experience banks (e.g., ReasoningBank): store lessons per trajectory, retrieve at test time. Retrieval relies on surface similarity and fails when test queries differ in wording. The agent gets fragmented, local advice rather than systematic global guidance.
• Sequential online editing (e.g., AutoSkill, EvoSkill): edit the skill once per trajectory. Sequential editing causes premature convergence to early-observed patterns, ignoring later diversity—like revising a book's outline after every chapter, so the final chapter's insights never reshape earlier structure.

The Three-Stage Pipeline

Stage 1: Trajectory generation. Run the agent on 200 tasks, producing success and failure trajectories. 200 trajectories on a 122B model take under 2 GPU-hours.

Stage 2: Parallel multi-agent patch proposal. Instead of one LLM reading all trajectories sequentially, dispatch 128 parallel "analyst sub-agents," each seeing one trajectory:

• Error Analyst (A⁻): uses multi-turn ReAct loops—inspecting files, comparing against ground truth, verifying fixes—until it finds the true root cause. Strict quality gate: trajectories without a verified root cause are discarded.
• Success Analyst (A⁺): single-turn call extracting generally valid patterns from success trajectories.

Key design: all analysts work on a frozen copy of the initial skill and cannot see each other, preventing premature convergence and preserving full diversity.

Stage 3: Conflict-free merging. All patches are merged simultaneously, not sequentially, in a layered (tournament-like) fashion:

• Layer 1: 32 patches merged into one intermediate patch
• Layer 2: intermediate patches merged again... until a final unified patch

The merge operator M is explicitly instructed: "if multiple patches independently propose similar edits, treat recurrence as evidence of a systematic property, prioritize it, and express it as a general principle."

This is inductive reasoning—extracting universal laws from specific observations. Not "task #47 failed," but "formulas must be recalculated after writing."

3. Why Parallel Merging Beats Sequential Editing

Direct comparison (Table 4):

| Method | 122B Vrf | 35B Vrf | Time | |--------|----------|---------|------| | Sequential B=4 | 59.00 | 26.17 | ~15 min | | Sequential B=1 | 61.83 | 26.00 | ~60 min | | Parallel (ours) | 65.83 | 27.00 | ~3 min |

Parallel significantly beats both sequential settings on 122B (+4.0 pp over B=1, +6.8 pp over B=4) at 1/20 the time.

Why? Sequential editing suffers from drift: every edit changes the skill, so subsequent analysts work on a modified version, creating dependencies and interference between patches. Parallel editing has all patches start from the same frozen initial skill, eliminating drift. Hierarchical merging then selects cross-trajectory recurring patterns via induction—patterns more likely to be systematic and transferable.

4. Results: A 35B Model Teaching the 122B Model

The most counterintuitive finding. Core results (Table 1, Avg column):

Skill Deepening (improving human-written skills):

• 122B-authored +Combined: +9.19 pp
• 35B-authored +Combined: +14.78 pp (the 35B's improved skill beats the 122B's own!)

Skill Creation (from scratch):

• 35B-authored +Error: +18.26 pp (best overall)
• WikiTQ (OOD): +57.65 pp—from 23.73% to 81.38%

Cross-model transfer: skills distilled by the 35B model lift the 122B model by +57.65 pp on WikiTQ; the reverse also yields significant gains for the 35B.

This means skill-distillation ability and task-execution ability are distinct dimensions. The 35B model executes DocVQA worse than the 122B, yet its distilled skills improve the 122B dramatically—inductive reasoning is an independent capability small models can master.

5. Honest Limits: Where Did 31% of Failures Go?

The paper analyzes failures extensively. 62 of 200 queries (31%) failed, for three reasons:

1. Navigation errors (38, 61%): the LLM chose the wrong top-level skill 2. Over-retrieval (19, 31%): correct topic found, but dragged along irrelevant documents 3. Synthesis errors (3, 5%): misread conditional instructions

This closely mirrors Corpus2Skill's failure modes—top-level classification granularity is the bottleneck. The authors deserve credit for not hiding failures but analyzing them.

6. Feynman-Style Scrutiny: Is This Really a "Skill"?

Q1: Naming ≠ understanding. The output is called a "skill," but is it a Markdown file, a set of procedures, or true craftsmanship—internalized expert intuition? From the paper's description, Trace2Skill produces a highly structured SOP: explicit steps, checklists, tool-selection rules. Human expert skill includes tacit knowledge—when to bend rules, when to follow them. More accurately, Trace2Skill produces an executable procedural specification, not a complete skill. But for agents, that suffices—agents don't need "feel," they need clear instructions.

Q2: Cargo cult? The form is complete: three-stage pipeline, parallel agent swarm, hierarchical merging, inductive reasoning. Do these forms deliver real improvement? The data says yes:

• Parallel merging beats sequential editing (Table 4)
• Agentic error analysis beats single LLM calls (Table 6, Avg +12.2 pp)
• A single skill document beats retrieval-style experience banks (Table 5, +13.8 pp Vrf)

Form has substance. Not cargo cult.

Q3: Selective reporting? A potential blind spot: all experiments use the Qwen3.5 family (122B and 35B). No validation on other families (Llama, GPT); cross-family transfer is untested. The equal-weighted "Avg" metric favors comprehensiveness but may mask significant failures in specific settings—e.g., 122B-authored Deepening +Success is -0.9 pp on Avg, which the paper honestly reports.

7. Practical Takeaways: Who Should Use This?

Good fit:

• A well-defined task domain (Excel processing, document QA)
• Ability to collect many trajectories (at least dozens to hundreds)
• Need for transferable, shareable skills (no retraining per model)

Poor fit:

• Rapidly changing domains (skills need frequent updates)
• No ground truth available to verify failure root causes
• Millisecond-level latency requirements (multi-turn agent analysis and merging are slow)

Relation to Corpus2Skill: Corpus2Skill solves knowledge-base navigation—turning static documents into navigable structure. Trace2Skill solves experience distillation—turning dynamic execution trajectories into reusable skills. They're complementary: Corpus2Skill helps the agent find the right knowledge; Trace2Skill helps it master the right skills.

8. Conclusion

Trace2Skill's most valuable contribution is shifting agent self-evolution from online sequential editing to offline parallel induction.

It proves one thing: the value of experience lies not in remembering more, but in distilling deeper. Parallel analysis of a few hundred trajectories yields more transferable, more general skills than sequential processing of each trajectory's local lessons.

And a 35B model suffices for the distillation—skill evolution doesn't require expensive proprietary models. Open models + many trajectories + the right distillation pipeline produce high-quality transferable skills.

> "Accumulate first, distill later"—future agents may no longer be "learn-while-doing" lifelong learners, but experts who "finish a batch of tasks, then sit down and write the manual."

---

📄 Paper: arXiv:2603.25158 🏢 Authors: Jingwei Ni, Yihao Liu, Xinpeng Liu, Yutao Sun, Mengyu Zhou, et al. (ETH Zurich / PKU / ZJU / Alibaba) 🔗 Code: https://github.com/trace2skill/trace2skill