Trace2Skill Deep Dive: Turning an Agent's Error Log into Transferable Skill Playbooks
> In one sentence: Trace2Skill demonstrates something counterintuitive — skills "written" by a small model (35B) can improve a large model (122B) by 57.65 percentage points. Not because it writes well, but because it induces correctly.
---
1. The Problem: Two Roadblocks to Skill Authoring
There are currently two dead ends for equipping agents with skills (Skills/MCP tools):
Manual authoring — slow, expensive, unscalable. Anthropic's official xlsx skill is a case in point: it helps the 122B model (48.33% verification pass rate), but actually *harms* the 35B model (dropping from 19% to 9.67%). Human-written skills are designed for the "strongest brain"; small models can't digest them.
Automatic learning — either fragmented (one skill per trajectory, unretrievable at inference time) or overfitted (sequential updates where later lessons overwrite earlier ones).
Trace2Skill's solution is simple: let the agent make lots of mistakes and successes first, then dispatch a swarm of "analysts" to read all trajectories simultaneously and induce a general-purpose SOP.
---
2. Architecture: A Three-Stage Pipeline
Stage 1: Trajectory Generation
- The agent runs on a task set with initial skills (human-written or LLM drafts)
- Collects both success (T⁺) and failure (T⁻) trajectories
- 200 trajectories, 50+ interaction turns, < 2 GPU-hours on the 122B model
- Each analyst works from a frozen copy of the initial skill, invisible to other analysts' patches
- Error analysts must provide causal verification — trajectories without an identified root cause are discarded
- All analysts follow Anthropic's skill-writing style (concise, actionable, progressive disclosure)
- Paper: J. Ni et al., "Trace2Skill: Distill Trajectory-Local Lessons into Transferable Agent Skills", arXiv:2603.25158, 2026
- GitHub: https://github.com/Qwen-Application/Trace2Skill
Stage 2: Parallel Multi-Agent Patch Proposals
This is Trace2Skill's core innovation:
| Analyst | Task | How it works | |---|---|---| | Success analyst A⁺ | Identify "what was done right" | Single-turn call, extracts generalizable correct patterns | | Error analyst A⁻ | Diagnose "why it failed" | ReAct multi-turn loop: inspect trajectory → read files → compare with ground truth → localize root cause |
Key constraints:
Stage 3: Conflict-Free Consolidation
All patches go through hierarchical merging (B_merge=32):
1. Deduplication: identical suggestions kept once 2. Conflict resolution: same-file same-scope edits are flagged as conflicts and retained 3. Inductive reasoning: identify prevalent patterns across patches, discard idiosyncrasies appearing in only a few 4. Format validation: references to nonexistent files are rejected outright
---
3. Core Insight: Induction Beats Retrieval
The fundamental difference from ReasoningBank: turn experience into skills, not memory entries.
| | ReasoningBank | Trace2Skill | |---|---|---| | Storage | Each trajectory's experience stored independently | Everything compressed into one skill | | Usage | Retrieve most relevant experience at inference | Directly preload the skill at inference | | Bottleneck | Retrieval quality depends on embedding alignment | No retrieval bottleneck | | 35B result | 20.50% (nearly useless) | 29.67% (significant gain) |
ReasoningBank fails on the 35B model because query representations and stored experience embeddings are misaligned — the small model's query vectors and the large model's stored experience don't share a semantic space, so retrieval returns irrelevant items.
Trace2Skill sidesteps this: distill experience into text-form skills, no vector retrieval needed.
---
4. The Most Striking Finding: Small Model Writes, Large Model Uses
The paper's most counterintuitive result:
| Skill author | Skill user | Scenario | Gain | |---|---|---|---| | Qwen3.5-35B | Qwen3.5-122B | Create+Error, WikiTQ | +57.65 pp | | Qwen3.5-122B | Qwen3.5-35B | Deepen+Error, Vrf | +27.00 pp |
Why do skills written by a 35B model dramatically help the 122B?
It's not that 35B "writes" better — it's that 35B's mistakes are more representative. Simple errors the 122B rarely makes, the 35B does make; the guardrails the 35B induces from its own failures happen to cover blind spots the 122B also has but hadn't surfaced.
This reveals a deeper principle: "inductive skill authoring" and "task execution" are two different capabilities. The 35B outperforms the 122B on DocVQA execution (0.6843 vs 0.6424 ANLS), yet is far weaker as a skill author than the 122B.
---
5. What the Learned SOPs Look Like
Top skills induced from 323 patches:
| Rank | SOP theme | Supporting patches | |---|---|---| | 1 | Formula recomputation and write-back verification | 178 | | 2 | Tool choice: openpyxl over pandas.to_excel() | 177 | | 3 | Explicit read-back verification (reopen after writing to confirm) | 138 | | 4 | Safe structural edits (delete rows in descending order, copy workbook first) | 53 | | 5–8 | Target range validation, data type preservation, etc. | 10–15 |
These aren't trajectory-specific hacks — they are cross-trajectory regularities. "177 patches all mention openpyxl is more reliable than pandas" is transferable knowledge.
---
6. Efficiency: Parallel vs Sequential
| Method | Time | 122B Vrf | 35B Vrf | |---|---|---|---| | Seq-B=1 (one at a time) | ~60 min | 61.83 | 26.00 | | Seq-B=4 (small batches) | ~15 min | 59.00 | 26.17 | | Parallel | ~3 min | 65.83 | 27.00 |
Parallel is ~20x faster *and* more effective, because: 1. Seeing all trajectories simultaneously enables cross-trajectory pattern identification 2. It avoids sequential-update "drift" — later lessons overwriting earlier ones
---
7. A Dialogue with Perplexity's Skill Philosophy
A previous post covered Perplexity's "three-layer tax" on skills (indexing tax, loading tax, runtime tax). Trace2Skill offers a complementary angle:
Perplexity asks: how do we keep skills from becoming a burden? Trace2Skill asks: how do we grow skills automatically from experience?
Together, the future of agent skills may look like: 1. Write thin by hand (Perplexity's philosophy — thin bootstrapping) 2. Run thick via agents (Trace2Skill's philosophy — automatic thickening from trajectories) 3. Inductive distillation (compress thick experience back into thin, transferable skills)
Trace2Skill proves this loop is feasible: a 35B-parameter model can do the distillation — no fine-tuning, no external retrieval — and the resulting skill transfers across model scales.
---
8. Limitations and Open Questions
1. Task complexity ceiling: tested on spreadsheets, VisionQA, and math reasoning; more complex multi-step tasks (e.g., software development) remain unvalidated 2. Trajectory generation cost: 200 trajectories × 50 turns × 122B model — although "only < 2 GPU-hours," still non-trivial upfront investment 3. Online learning integration: currently batch-generate-then-distill; continuous online settings unexplored 4. Automatic skill conflict resolution: conflict detection is programmatic; more complex semantic conflicts still need human intervention
---
Key Takeaways
1. Induction > retrieval: distilling experience into a text skill beats storing it as retrieval memory 2. Parallel > sequential: analyzing all trajectories at once identifies prevalent patterns and is faster too 3. Small models can write skills for large models: induction and execution are different capabilities — the 35B's "error notebook" is more valuable to the 122B 4. No parameter updates, no external retrieval, 35B suffices: a surprisingly low barrier to entry
---
References