SkillAA: Attribution-Guided Skill-Graph Updating with Targeted Validation and Rollback

The Problem: Failure Doesn't Always Mean "Fix the Skill"

Researchers at Nanjing University (Ziqiao Shang, Lingyue Ge, Lan-Zhe Guo) identified a blind spot in current LLM skill-evolution systems: they all map "failure → modify skill" through a single channel. In reality, failures have diverse causes—wrong skill content, missed retrieval evidence, visual recognition errors, or an overly strict scorer. Blaming every failure on the skill graph is like blaming every bug on the code: sometimes it's hardware, data, or contradictory requirements.

SkillAA replaces this with an engineering pipeline of precise attribution + targeted repair + gated validation.

Core Design: Skills as an Addressable Graph

SkillAA turns skills from flat text into a directed graph. Each node has three structured fields:

• when_to_use: trigger conditions
• how_to_use: execution steps
• avoid: exclusion boundaries

Nodes connect via typed edges such as prereq and enhance. The key benefit: every editable location has an address—like src/auth/login.py:42—so you can point to "this field of this node" instead of "rewrite the whole module." That is what "Attribution" means in the name.

Six Attribution Paths (Including NO_PATCH)

When execution fails, a teacher model assigns one of six codes:

| Code | Failure condition | Authorized modification | |---|---|---| | MISSING_SKILL_FAMILY | Whole skill missing | Add node + edges | | MISSING_ACTIVATION_CUE | Skill exists but not activated | Extend when_to_use only | | HARMFUL_EXISTING_RULE | Existing semantics cause errors | Replace only the offending field | | MISSING_OR_INCORRECT_PROCEDURE | Missing steps/boundaries/relations | Complete how_to_use or avoid | | EXECUTION_LAPSE | Graph correct, model deviated | NO_PATCH | | INSUFFICIENT_EVIDENCE | Cannot localize the fault | NO_PATCH |

NO_PATCH is a first-class output: when the graph is right (model mis-executed) or evidence is insufficient, the correct action is to change nothing. Forcing edits would degrade good skills.

Two Gates: Transaction-Style Skill Management

Local Gate

For each candidate edit (or atomic group), SkillAA re-runs potentially affected cases (Q_use, Q_src, Q_unk) and counts n_fixed, n_broken, n_unresolved. An edit is kept only if n_fixed > n_broken + n_unresolved; otherwise the entire atomic group rolls back—like row-level constraints in a database.

Big Gate

All accepted edits merge into a candidate graph \(\tilde{G}^{(t+1)}\), evaluated on the whole update pool. Commit only if n_corrected > n_regressed; otherwise both the graph and reference execution traces revert to the epoch's start—like a database COMMIT.

Key Experiments

On SearchQA, LiveMath, and DocVQA with gpt-5.6-sol as student and teacher over three epochs:

• SearchQA: 81.5%
• LiveMath: 66.7%
• DocVQA: 91.2%

SkillAA achieved the highest point estimates across nine model-benchmark combinations. Two negative findings are more interesting than the numbers:

Negative Result 1: ALFWorld

Under a 50-step budget on ALFWorld, the initial skill graph performed best—SkillAA's edits dropped success from 95.5% to 92.5%. When initial skills already suffice and residual failures come from exploration or execution, editing the graph actively hurts.

Negative Result 2: Residual Error Audit

Manual audits of six held-out errors revealed four different required fixes: 1. Scorer issue: "Manuel Noriega" vs. gold "Noriega"—fix the scorer, not the skill 2. Missing evidence: retrieval missed "first" in "first Secretary of State"—fix retrieval 3. Theorem misidentification: add a domain knowledge base, not skills 4. Visual recognition error: handwritten 7/18 read as 11/18—add OCR cross-validation

Only one error was a genuine skill defect, and even it didn't qualify for a persistent patch—repeated occurrences are needed to confirm a systematic flaw. "Wrong answer" does not equal "wrong skill."

Engineering Insights

1. Make skills addressable: splitting skills into when_to_use / how_to_use / avoid fields within a graph converts "readable documents" into "addressable codebases," enabling targeted fixes and clean rollbacks.

2. NO_PATCH is legitimate: respecting failure diversity prevents blindly pushing every failure into the skill system—like prescribing antibiotics for viral infections.

3. Gates = transactional management: Local Gate as row-level constraints, Big Gate as transaction commit, catching regressions where a skill fix helps task A but breaks task B.

Graph-Addressable vs. Capability-Limited Failures

The paper proposes a framework distinguishing:

• Graph-addressable failures: fixable by editing a field or edge
• Capability-limited failures: the graph is right but the model lacks ability

Repeated Big Gate rejections signal that graph-fixable failures are exhausted and model upgrades are needed. Future systems should alternate: edit the graph, then upgrade the model, iteratively.

Practical Takeaways for Agent Builders

1. Give your skills addresses: use when_to_use / how_to_use / avoid fields in a graph structure instead of monolithic markdown. 2. Set two gates: run regression tests before committing skill changes; only submit when global metrics are net-positive. 3. Learn to say NO_PATCH: diagnose first—trigger miss or wrong step means edit the graph; insufficient capability means change the model.

Open Source

Code: https://github.com/Ziqiao-Shang/SkillAA — includes the graph optimizer, gate implementations, benchmark data preparation scripts, and full per-example prediction records. Terminal graphs (SearchQA: 30 nodes / 8 edges; LiveMath: 37 nodes / 40 edges; DocVQA: 20 nodes / 19 edges) are released for inference-only ablations.

Paper Info

• Title: SkillAA: Attribution-Guided Skill-Graph Updating with Targeted Validation and Rollback
• Authors: Ziqiao Shang, Lingyue Ge, Lan-Zhe Guo (Nanjing University)
• arXiv: 2609.20455
• Code: https://github.com/Ziqiao-Shang/SkillAA

Closing thought: SkillAA's most admirable quality is its respect for failure. Knowing when *not* to change is harder—and more important—than knowing what to change. NO_PATCH as a first-class output is a design philosophy every self-evolving agent team should borrow: not every failure is the skill's fault, and admitting that is what makes skill systems truly reliable.