Trace2Skill: Turning an Agent's Error Notebook into Transferable Skills
> One-liner: Trace2Skill demonstrates a counter-intuitive result — skills authored by a small (35B) model can lift a large (122B) model by 57.65 percentage points. Not because the small model writes better, but because it generalizes better.
---
1. The Problem: Two Mountains in Skill Authoring
Equipping agents with skills (Skill / MCP tool) currently follows two dead ends:
- Manual authoring — slow, costly, not scalable. Anthropic's official xlsx skill, for example, helps the 122B model (48.33% validation pass rate) but actually hurts the 35B model (drops from 19% to 9.67%), because human skills are designed for the strongest minds and are indigestible for smaller ones.
- Automatic learning — either fragmented (one skill per trajectory, so retrieval fails to find anything relevant) or overfitting (sequential updates where later patches overwrite earlier ones).
- The agent runs the initial skill (human-written or LLM-drafted) on a task set.
- Two trajectory types are collected: success (T⁺) and failure (T⁻).
- 200 trajectories × 50+ interactive rounds with a 122B model require < 2 GPU-hours.
- Each analyst operates against a frozen copy of the initial skill and cannot see other analysts' patches.
- Error analysts must perform causal verification — trajectories without an identifiable root cause are discarded.
- All analysts follow Anthropic's skill-writing style: concise, actionable, layered.
- Perplexity asks: How do we keep skills from becoming a burden?
- Trace2Skill answers: How do skills grow automatically from experience?
- Paper: J. Ni et al., *Trace2Skill: Distill Trajectory-Local Lessons into Transferable Agent Skills*, arXiv:2603.25158, 2026
- GitHub: https://github.com/Qwen-Application/Trace2Skill
Trace2Skill's answer is simple: let the agent fail and succeed extensively, then dispatch a team of "analysts" to read all trajectories in parallel and induce general-purpose Standard Operating Procedures (SOPs).
---
2. Architecture: A Three-Stage Pipeline
Stage 1 — Trajectory Generation
Stage 2 — Parallel Multi-Agent Patch Proposal
This is Trace2Skill's core innovation:
| Analyst | Task | Mode | |---|---|---| | Success analyst A⁺ | Identify what worked | Single-turn call, extracts generalizable correct patterns | | Error analyst A⁻ | Diagnose what failed | ReAct multi-turn loop: inspect trajectory → read files → compare against ground truth → locate root cause |
Key constraints:
Stage 3 — Conflict-Free Integration
All patches are merged hierarchically (B_merge = 32):
1. Deduplication — identical suggestions are kept only once. 2. Conflict resolution — same-file, same-range edits are flagged and preserved. 3. Inductive reasoning — prevalent patterns across many patches are kept; patch-specific noise is discarded. 4. Format validation — references to non-existent files are rejected outright.
---
3. Core Insight: Inductive Reasoning > Retrieval Memory
The fundamental difference from ReasoningBank: experience becomes a skill, not a memory bank.
| | ReasoningBank | Trace2Skill | |---|---|---| | Storage | Per-trajectory experience, stored independently | All trajectories compressed into one skill | | Use | Retrieve the most relevant experience at inference | Pre-load the skill at inference | | Bottleneck | Retrieval quality depends on embedding alignment | No retrieval bottleneck | | 35B result | 20.50% (nearly useless) | 29.67% (significant gain) |
ReasoningBank fails on 35B because query representations and stored experience embeddings are not aligned — small-model queries live in a different semantic space than large-model-stored experience, so retrieved items are irrelevant.
Trace2Skill sidesteps this entirely: experience is distilled into text skills, with no vector retrieval required.
---
4. The Most Striking Finding: Small Model Writes, Large Model Uses
The paper's most counter-intuitive result:
| Skill author | Skill user | Scenario | Gain | |---|---|---|---| | Qwen3.5-35B | Qwen3.5-122B | Creation+Error, WikiTQ | +57.65 pp | | Qwen3.5-122B | Qwen3.5-35B | Deepening+Error, Vrf | +27.00 pp |
Why can a 35B-written skill lift a 122B so dramatically?
Not because 35B writes better — but because 35B's mistakes are more representative. Simple errors that 122B rarely makes are exactly the errors 35B makes; the guardrails 35B induces from its own failures happen to cover blind spots that 122B also has but never noticed.
This exposes a deeper principle: "inductive skill authoring" and "task execution" are different capabilities. On DocVQA, 35B actually executes better than 122B (0.6843 vs 0.6424 ANLS), yet as a skill author it is far weaker than 122B.
---
5. What the Learned SOPs Look Like
Top skills induced from 323 patches:
| Rank | SOP topic | Supporting patches | |---|---|---| | 1 | Formula recalculation and write-back verification | 178 | | 2 | Tool choice: openpyxl beats pandas.to_excel() | 177 | | 3 | Explicit read-back verification (re-open after write to confirm) | 138 | | 4 | Structural-edit safety (delete rows in descending order; copy workbook first) | 53 | | 5–8 | Target-range validation, data-type preservation, etc. | 10–15 |
These are not trajectory-specific hacks but laws common across many trajectories — e.g., "177 patches mention that openpyxl is more reliable than pandas" is genuinely transferable knowledge.
---
6. Efficiency: Parallel vs. Sequential
| Method | Time | 122B Vrf | 35B Vrf | |---|---|---|---| | Seq-B=1 (single, sequential) | ~60 min | 61.83 | 26.00 | | Seq-B=4 (small batch) | ~15 min | 59.00 | 26.17 | | Parallel | ~3 min | 65.83 | 27.00 |
Parallel is not only ~20× faster, it is also more accurate, because: 1. Seeing all trajectories at once reveals cross-trajectory prevalent patterns. 2. It avoids the "drift" of sequential updates, where later patches overwrite earlier ones.
---
7. Dialogue with the Perplexity Skill Philosophy
A prior essay interpreted Perplexity's skill "three-layer tax" (index tax, load tax, runtime tax). Trace2Skill adds a complementary perspective:
Together, the future of agent skills may be: 1. Thin human write (Perplexity — thin bootstrap). 2. Agent runs thick (Trace2Skill — automatic thickening from trajectories). 3. Inductive distillation (compress thick experience back into a thin, transferable skill).
Trace2Skill shows this loop is feasible: a 35B model is sufficient to perform "inductive distillation", no fine-tuning required, no external retrieval needed, and the resulting skills transfer across model scales.
---
8. Limitations and Open Questions
1. Task-complexity ceiling — evaluated on spreadsheets, VisionQA, and math reasoning; more complex multi-step tasks such as software development remain untested. 2. Trajectory-generation cost — 200 trajectories × 50 rounds × 122B still amounts to a non-trivial upfront investment, even at < 2 GPU-hours. 3. Online learning — currently batch-then-distill; online continual-learning scenarios are unexplored. 4. Automatic conflict resolution — conflict detection is programmatic; deeper semantic conflicts still require human intervention.
---
Core Conclusions
1. Induction > retrieval — distilling experience into text skills is more efficient than storing it as retrievable memory. 2. Parallel > sequential — analyzing all trajectories simultaneously reveals cross-trajectory prevalent patterns, and is dramatically faster. 3. A small model can write skills a large model uses — because induction and execution are different capabilities; the 35B "error notebook" is more valuable to the 122B than its own. 4. No parameter updates, no external retrieval, 35B parameters is enough — the bar is surprisingly low.
---
References