WikiSkill Explained: Compiling Agent Experience into Persistent Knowledge for Skill Evolution

Forum topic · 小凯 · 2026-08-28

Summary

WikiSkill (arXiv:2608.27454, Tang et al., 2026) is a framework for AI agent skill evolution that addresses a key weakness of prior methods: experience gathered during one evolution round is discarded before the next round begins. The system organizes information into three layers: a raw layer of immutable execution trajectories, a persistent wiki layer that is append-only (containing patterns, an evolution log, and an impact tracker), and a skill layer holding executable SKILL.md guides that can be rolled back if validation degrades. A four-step loop alternates between an inference agent (which cannot see the wiki), a wiki maintainer that performs root-cause analysis on sampled traces, a skill proposer, and gated validation with rollback. Evaluated on GSM8K, WebShop, SpreadsheetBench, MuSiQue, and ALFWorld across five models (Qwen-3.5-4B/9B, Qwen-3.6-27B, Gemma-3-4B, Gemini-2.0-Flash), WikiSkill consistently outperforms no-skill baselines and prior methods such as EvoSkill and SkillOpt. Gains grow with model scale (12.3% to 23.9%), a 9B model with WikiSkill surpasses a 27B model without it on ALFWorld, and skills transfer across models. Ablations confirm the persistent wiki is essential, supporting the paper's core thesis that knowledge must be retained even when skills are reverted.

From Experience to Wisdom: A Deep Dive into WikiSkill

This post is an English translation/summary of a Chinese forum article analyzing the paper WikiSkill: Compiling Agent Experience into Persistent Knowledge for Skill Evolution (Liyan Tang, Cyrus Rashtchian, Chun-Sung Ferng, Andrew Tomkins, Da-Cheng Juan, Tu Vu), arXiv:2608.27454, released 2026-08-27.

The Problem: AI Keeps Falling Into the Same Pit

Skill evolution methods (e.g., EvoSkill, Trace2Skill, SkillOpt) let agents distill reusable skills from exploration. However, each round of evolution is treated as an independent event: failed trajectories, rejected proposals, and near-misses are lost before the next round. The system "re-learns" the same lessons repeatedly. WikiSkill asks: can agent experience be accumulated persistently so each generation of skills builds on prior knowledge?

Three-Layer Architecture

Raw layer — an archive of all observations, actions, tool calls, and reasoning from every task execution. Append-only, like medical records.
Wiki layer — the core innovation: a continuously accumulating, never-rolled-back knowledge base consisting of:
patterns/: a growing catalog of failure modes, effective strategies, and reusable workarounds, written as structured Markdown docs (what problem → how it was found → how it was solved).
logs.md: an evolution log tracking accepted/rejected skill proposals and the reasons.
skill-impact.md: an impact tracker for recurring errors and effective interventions.
Key principle: the skill layer can be rolled back, but the wiki layer only moves forward. Even if a candidate skill is rejected, its lessons stay in the wiki.
Skill layer — the active, executable knowledge the agent actually uses. Each skill is a folder with a SKILL.md guide and a PURPOSE.md tracing the skill back to wiki patterns. Skills are conditional and reversible: if a new skill set degrades validation performance, the layer rolls back.

The Four-Step Evolution Loop

1. Inference agent executes tasks using only the skill layer — by design it cannot see the wiki (exposing it made agents overly conservative and harmed skill development). 2. Wiki maintainer samples success/failure trajectories from the raw layer, performs root-cause analysis, and incrementally updates the wiki. 3. Skill proposer reads the updated wiki plus recent traces and uses ReAct-style reasoning to create, modify, or delete skills. 4. Gating & rollback — candidate skills are validated; improvements are accepted, regressions trigger a rollback. The wiki retains everything regardless of outcome.

Experimental Results

Evaluated on five benchmarks (math reasoning GSM8K, web search WebShop, spreadsheets SpreadsheetBench, long-document QA MuSiQue, embodied tasks ALFWorld) across five models (Qwen-3.5-4B, Qwen-3.5-9B, Qwen-3.6-27B, Gemma-3-4B, Gemini-2.0-Flash):

Consistent wins: WikiSkill beats the no-skill baseline and prior methods (EvoSkill, SkillOpt) on every benchmark and model.
Gains scale with model size: +12.3% for Qwen-3.5-4B, +17.5% for Qwen-3.5-9B, +23.9% for Qwen-3.6-27B.
Small beats big: Qwen-3.5-9B with WikiSkill (47.4%) outperforms Qwen-3.6-27B without skills (39.4%) on ALFWorld.
Cross-model transfer: skills evolved by Qwen-3.6-27B gave Qwen-3.5-9B 70.2% vs. 63.4% with self-evolved skills — a larger model's distilled "teaching experience" can be more valuable than self-evolution.
Ablations confirm the persistent wiki is critical: without it, skill evolution degrades substantially.

Deeper Implications

The counterintuitive design philosophy: skills may be revoked, but knowledge must be kept. Learning is split into two independent processes — experience accumulation (wiki) and skill iteration (skill layer). This separation also yields interpretability: every skill's PURPOSE.md traces its origin to wiki patterns, unlike black-box end-to-end training.

The architecture naturally extends to multi-agent settings — specialist agents (e.g., medical imaging, pathology) maintaining private wikis while sharing a common-knowledge wiki — pointing toward structured exchange of experience rather than mere parameter sharing.

Reference

Tang, L., Rashtchian, C., Ferng, C. S., Tomkins, A., Juan, D. C., & Vu, T. (2026). WikiSkill: Compiling Agent Experience into Persistent Knowledge for Skill Evolution. *arXiv preprint arXiv:2608.27454*.