SkillAA: When AI Learns to Say "This Isn't the Skill's Fault" — Precise Attribution and Rollback for Skill Graphs

An Awkward Scenario

Imagine you are a senior engineer with an intern named G. G is smart but can't remember process details, so you write him an "operations manual" — when to query which database, when to call which tool, when to skip which step.

One day, G fails on a QA task. You review his execution logs and find the problem in step three: he should have looked up "the leader convicted by Panamanian courts in 1993," but answered Manuel Noriega instead of the gold answer "Noriega."

What would you do?

A. Rewrite the whole manual B. Add a rule: "use short names for people" C. Fix the grader so it recognizes aliases D. Change nothing, because it's a grader problem

If you chose A or B, congratulations — you're making the same mistake most LLM skill-optimization systems make today.

The Root Problem: Failure ≠ "The Skill Should Change"

The SkillAA team at Nanjing University (Ziqiao Shang, Lingyue Ge, Lan-Zhe Guo) identified an overlooked problem: all current "skill evolution" systems use a single mapping of "failure → edit skill." But real-world failures have diverse causes — a miswritten skill, retrieval missing key evidence, the model misreading characters visually, or simply an overly strict grader.

Attributing every failure to "the skill is wrong" is like blaming every bug on "bad code" — sometimes it's a hardware fault, sometimes missing data, sometimes contradictory requirements. Blindly editing code only makes things worse.

SkillAA's core contribution is an engineering pipeline of precise attribution + targeted repair + gated validation for skill optimization.

Core Design: Making Skills an "Addressable Graph"

SkillAA's first key decision is to turn skills from flat text into an addressable directed graph.

Each node is split into three structured fields:

• when_to_use: trigger conditions
• how_to_use: concrete execution steps
• avoid: exclusion boundaries

Nodes connect via typed edges, e.g., prereq (dependency) and enhance (composition).

The benefit: every editable location has an address. Like src/auth/login.py:42 in a codebase — when a bug appears, you can point at the exact line/branch rather than "rewrite the whole login module."

That's the real meaning of "Attribution" in SkillAA's name: not vaguely saying "the skill is wrong," but locating the exact field of a specific node.

Six Attribution Paths: A Taxonomy of Failure

SkillAA defines six attribution paths. When execution fails, the teacher model diagnoses the trace and assigns one of six codes:

| Attribution code | Failure condition | Authorized modification | |---|---|---| | MISSING_SKILL_FAMILY | Entire skill missing | Add node + bind edges | | MISSING_ACTIVATION_CUE | Skill exists but wasn't activated | Only extend when_to_use | | HARMFUL_EXISTING_RULE | Existing semantics cause errors | Only replace the problematic field | | MISSING_OR_INCORRECT_PROCEDURE | Missing steps/boundaries/relations | Complete how_to_use or avoid | | EXECUTION_LAPSE | Graph is correct; model didn't follow it | NO_PATCH | | INSUFFICIENT_EVIDENCE | Evidence insufficient to localize | NO_PATCH |

Note the last two rows. NO_PATCH is a first-class output in SkillAA — when diagnosis shows "the graph is right, execution drifted" or "evidence is insufficient to localize," the correct action is to change nothing.

This seems simple but is profound. All current skill-evolution systems implicitly assume "if it failed, change something." SkillAA explicitly rejects that assumption: some failures aren't the skill's fault, and forcing edits corrupts good skills.

Two Gates: Transaction-Like Skill Management

After localization, SkillAA doesn't commit candidate edits directly. Two gates screen them, like database transaction isolation.

Local Gate: Localized Regression Testing

For each candidate edit (or a mutually dependent atomic group), SkillAA re-runs only potentially affected cases:

• Cases that used this node (Q_use)
• Cases that motivated the edit (Q_src)
• Cases with missing records (Q_unk)

After re-running, it counts:

• n_fixed: previously wrong, now correct
• n_broken: previously correct, now wrong
• n_unresolved: still wrong

An edit is kept only if n_fixed > n_broken + n_unresolved. Otherwise the entire atomic group rolls back — node fields restored, added nodes and edges revoked.

Big Gate: Epoch-Level Commit

All Local-Gate-approved edits merge into a candidate graph \(\tilde{G}^{(t+1)}\), evaluated on the whole update pool:

• n_corrected: fixed cases
• n_regressed: broken cases

The candidate graph is committed only if n_corrected > n_regressed. Otherwise, both the graph and reference traces roll back to the epoch's start state.

Key Experiments: Not All Failures Are "Graph-Fixable"

SkillAA was validated on three benchmarks: SearchQA (QA), LiveMath (math), DocVQA (document visual QA). With gpt-5.6-sol as both student and teacher, after three epochs:

• SearchQA: 81.5%
• LiveMath: 66.7%
• DocVQA: 91.2%

SkillAA achieved the highest point estimates across nine model-benchmark combinations. But the more interesting findings are two "negative" results.

Negative Result 1: Editing Hurts on ALFWorld

On ALFWorld (interactive household tasks), all methods reach 100% within 100 steps. But under a tight 50-step budget, the initial skill graph is the best — after SkillAA's edits, success dropped from 95.5% to 92.5%.

A counterintuitive but crucial finding: when initial skills already suffice and remaining failures come from exploration or execution, editing not only doesn't help — it hurts.

Like over-optimizing a correct function and introducing bugs. Knowing when not to change matters as much as knowing what to change.

Negative Result 2: Residual Error Audit

The team manually audited errors on six held-out cases, finding four entirely different required fixes:

1. Grader issue: model answered "Manuel Noriega," gold was "Noriega" — fix the grader, not the skill 2. Missing evidence: retrieval missed "first" in "first Secretary of State" — fix retrieval, not the skill 3. Theorem misidentification: a related but non-equivalent theorem was chosen — add a domain knowledge base, not skill edits 4. Visual misrecognition: handwritten 7/18 read as 11/18 — add OCR cross-validation, not skill edits

Only one of six errors was a genuine skill defect (role selection) — and appearing only once, it didn't even qualify for a persistent edit: systematic skill defects require repeated occurrences of the same error class to confirm.

This residual audit is the paper's most valuable engineering contribution. It tells us: "wrong answer" ≠ "wrong skill." Funneling all errors into the skill system is like blaming every system failure on the application layer.

Engineering Insights: Three Things That Matter More Than "Editing Skills"

Insight 1: Make Skills an "Addressable Space"

Going from "a block of skill text" to "a graph of node fields and typed edges" isn't just a representation upgrade — it turns skills from "readable documents" into "an addressable codebase."

Without addresses you can only say "this skill has a problem." With addresses you can say "node #7's when_to_use field is missing its third trigger." The finer the granularity, the more precise the repair, the cleaner the rollback.

Insight 2: NO_PATCH Is a First-Class Citizen

SkillAA goes against the grain of "edit more, edit better" by elevating "don't edit" to equal importance.

This isn't conservatism — it's respect for the diversity of failure causes. When failure can come from retrieval, perception, reasoning, execution, or the grader, blaming everything on skills is like prescribing antibiotics for every illness.

NO_PATCH acts as a triage desk: first determine whether the failure is skill-fixable; if not, refer it to other components.

Insight 3: Two Gates = Transactional Skill Management

Local Gate + Big Gate essentially transplant database transaction management into skill optimization:

• Local Gate = row-level constraints (edits can't break local consistency)
• Big Gate = transaction commit (an epoch's edits must be globally net-positive)

This solves a long-ignored problem: side effects of skill edits. Editing one skill can fix task A while breaking task B — nobody checked this before. SkillAA's Local Gate specifically catches such regressions.

A Deeper Insight: When to Edit the Graph vs. When to Change the Model

The paper ends with an elegant framework: graph-addressable vs. capability-limited failures.

• Graph-addressable failure: a specific field or edge is defective; editing fixes it
• Capability-limited failure: the graph is right, but the model itself lacks capability (reasoning, perception, execution)

When epoch-level edits repeatedly get rejected by the Big Gate, that's a stop signal — not that the model has peaked, but that "everything graph-fixable has been fixed; the rest requires model changes."

Future systems should alternate: edit the graph for graph-addressable failures, improve the model for capability-limited ones, iterating between the two.

Practical Takeaways for Agent Builders

1. Give your skills addresses: don't write skills as a blob of markdown. Split into when_to_use / how_to_use / avoid fields and manage dependencies as a graph, so failures localize to a specific field instead of "rewrite the whole skill."

2. Set up two gates: before editing, ask "which passing cases does this affect?" After editing, run regression tests. Commit only if globally net-positive; otherwise roll back.

3. Learn to say NO_PATCH: not every failure warrants a skill edit. Diagnose first — missed trigger? wrong procedure? or insufficient model capability? Edit the graph for the first two; change the model for the last.

Open-Source Code

Code: https://github.com/Ziqiao-Shang/SkillAA

The repo includes the graph optimizer, Gate implementations, benchmark data preparation scripts, and complete per-example prediction records. With gpt-5.6-sol as student/teacher, three epochs reproduce the main-table results. Terminal graphs (SearchQA: 30 nodes / 8 edges; LiveMath: 37 nodes / 40 edges; DocVQA: 20 nodes / 19 edges) are released for inference-only ablations.

Paper Info

• Title: SkillAA: Attribution-Guided Skill-Graph Updating with Targeted Validation and Rollback
• Authors: Ziqiao Shang, Lingyue Ge, Lan-Zhe Guo (Nanjing University)
• arXiv: 2609.20455
• Code: https://github.com/Ziqiao-Shang/SkillAA

Personal take: What's most admirable about SkillAA isn't the numbers but its respect for "failure." Current AI skill-evolution systems chase "edit more, edit faster," but mature engineering systems know: knowing when not to change is harder and more important than knowing what to change. NO_PATCH as a first-class output is a design philosophy every self-evolving agent team should borrow.