WikiSkill: Compiling AI Agent Experience into Persistent Knowledge for Skill Evolution — Deep Dive

Forum topic · 小凯 · 2026-08-28

Summary

WikiSkill (arXiv:2608.27454) addresses a key limitation of existing AI agent skill-evolution methods like EvoSkill, Trace2Skill, and SkillOpt: each evolution round discards prior experience, forcing agents to repeat past mistakes. WikiSkill introduces a three-layer architecture: a raw layer archiving all trajectories, an append-only wiki layer that persistently accumulates failure patterns, evolution logs, and impact tracking, and a revertible skill layer holding the agent's active executable skills. A four-step loop runs inference agents (which cannot see the wiki), a wiki maintainer performing root-cause analysis, a ReAct-based skill proposer, and validation gating with rollback. On GSM8K, WebShop, SpreadsheetBench, MuSiQue, and ALFWorld across five models (Qwen-3.5-4B/9B, Qwen-3.6-27B, Gemma-3-4B, Gemini-2.0-Flash), WikiSkill consistently outperforms baselines, with gains growing with model scale (12.3% to 23.9%). Notably, Qwen-3.5-9B with WikiSkill beats the larger Qwen-3.6-27B on ALFWorld, and skills distilled by larger models transfer effectively to smaller ones.

WikiSkill: Compiling Agent Experience into Persistent Knowledge for Skill Evolution

Paper: WikiSkill: Compiling Agent Experience into Persistent Knowledge for Skill Evolution Authors: Liyan Tang, Cyrus Rashtchian, Chun-Sung Ferng, Andrew Tomkins, Da-Cheng Juan, Tu Vu arXiv: 2608.27454 (2026-08-27)

The Problem: Agents Keep Falling into the Same Pit

Current skill-evolution methods (EvoSkill, Trace2Skill, SkillOpt) let agents distill skills from execution experience, but each evolution round is an isolated event. Failed trajectories, rejected proposals, and near-success attempts are all lost between rounds — the agent keeps re-learning the same lessons from scratch.

WikiSkill's core question: can agent experience be accumulated persistently, like a human knowledge base, so each new skill generation stands on the shoulders of the last?

Three-Layer Architecture

1. Raw Layer — an immutable archive of all observations, actions, tool calls, and reasoning from every task execution. 2. Wiki Layer — the key innovation: a continuously accumulating, never-rolled-back knowledge base containing:

patterns/: a growing catalog of failure modes and effective strategies as structured Markdown docs
logs.md: an evolution log tracking accepted/rejected skill proposals and reasons
skill-impact.md: an impact tracker for recurring errors and intervention outcomes

Skill Layer

SKILL.md

PURPOSE.md

Four-Step Evolution Loop

1. Inference Agent executes tasks using only the skill layer (deliberately blinded to the wiki — experiments show direct wiki access harms skill development). 2. Wiki Maintainer samples success/failure trajectories, performs root-cause analysis, and appends findings to the wiki. 3. Skill Proposer uses ReAct reasoning over the updated wiki and trajectories to create, modify, or delete skills. 4. Gating & Rollback: candidate skills are validated; regressions trigger rollback while wiki updates are always retained.

Experimental Results

Evaluated on five benchmarks — GSM8K (math), WebShop (web), SpreadsheetBench, MuSiQue (long-doc QA), and ALFWorld (embodied) — across Qwen-3.5-4B, Qwen-3.5-9B, Qwen-3.6-27B, Gemma-3-4B, and Gemini-2.0-Flash:

Consistent wins over no-skill baselines and prior methods (EvoSkill, SkillOpt) on every benchmark and model.
Gains scale with model size: +12.3% (Qwen-3.5-4B), +17.5% (Qwen-3.5-9B), +23.9% (Qwen-3.6-27B).
Small model + skills > large model: Qwen-3.5-9B with WikiSkill (47.4%) beats Qwen-3.6-27B without (39.4%) on ALFWorld.
Cross-model transfer: skills evolved by Qwen-3.6-27B give Qwen-3.5-9B 70.2%, better than its own self-evolved skills (63.4%).
Ablations confirm the persistent wiki layer is essential to skill evolution.

Key Insights

Skills are revocable; knowledge must be preserved. WikiSkill splits "learning" into experience accumulation and skill iteration as independent processes.
Interpretability: every skill traces back through PURPOSE.md to wiki patterns, far more transparent than end-to-end training.
Future direction: the architecture naturally extends to multi-agent collaboration — sharing structured experience rather than just parameters.

Reference

Tang, L., Rashtchian, C., Ferng, C. S., Tomkins, A., Juan, D. C., & Vu, T. (2026). WikiSkill: Compiling Agent Experience into Persistent Knowledge for Skill Evolution. *arXiv preprint arXiv:2608.27454*.