SkillCoach: Process Auditing for Agent Skill Use — From Outcome Correctness to Process Quality

Paper: SkillCoach: Self-Evolving Rubrics for Evaluating and Enhancing Agentic Skill-Use Authors: Jiayin Zhu, Kelong Mao, Yudong Guo, Dengbo He, Sulong Xu, Simiu Gu, Yutao Yue Paper link: https://arxiv.org/abs/2607.01874 TL;DR: Stop judging agents only by whether the task was completed — SkillCoach performs four-dimensional process audits to catch hidden "lucky success" errors, and can also be used in reverse to filter high-quality training data.

1. The Blind Spot in Agent Evaluation: "Lucky Success" Is More Dangerous Than Failure

Current AI agent evaluation has a fundamental flaw: it only looks at the final result.

Task done? Mark as success. Task failed? Mark as failure. This binary evaluation is adequate for simple scenarios, but in real skill-library environments it misses a large amount of critical information.

Four ways an agent can "get lucky":

1. Selecting a distractor skill — the skill library contains semantically similar overlapping skills; the agent picks the wrong one but works around the problem via trial and error 2. Skipping critical steps — an execution happens to skip a required check, but the result is still correct because the input data coincidentally satisfies the assumptions 3. Composing workflows incorrectly — dependencies between skills are mishandled, but by chance this doesn't affect the outcome for a specific input 4. Omitting final verification — the output validation required by the skill is never executed, but external factors make the result correct anyway

All four cases are labeled "success" under traditional evaluation, yet they represent unreliable skill use. Worse, if such data is used to train agents, wrong skill-usage patterns get baked into the model.

2. SkillCoach's Core Idea: Process Quality ≠ Outcome Success

SkillCoach shifts agent skill-use evaluation from outcome-oriented to process-oriented.

A Four-Dimensional Capability Assessment System

SkillCoach defines agent skill use as a trajectory-level meta-capability, scored across four dimensions:

1. Skill Selection

• Checks whether the agent selected the correct skills
• Penalizes missing gold-standard skills
• Penalizes selecting distractor skills (semantically similar but functionally different)
• Uses F1 score to jointly assess precision and recall

2. Skill Following

• Checks whether the agent fully executed the steps required by the skill
• Verifies coverage of critical steps
• Penalizes skipping required operations

3. Skill Composition

• Checks whether cross-skill dependencies are handled correctly
• Verifies logical consistency of workflow composition
• Penalizes incorrect skill chaining

4. Skill-Grounded Reflection

• Checks whether the agent performed the output verification required by the skill
• Verifies the agent actively validates result correctness
• Penalizes "run and forget" behavior without reflection

Key design: keep the external verifier as an independent signal

SkillCoach does not replace the final verifier; it runs in parallel with it:

• Process score: four-dimensional capability score (SkillCoach)
• Outcome signal: whether the task ultimately succeeded (external verifier)

This splits trajectories into four categories:

• ✅ Good process + correct result → high-quality sample
• ⚠️ Poor process + correct result → "lucky success" sample (dangerous!)
• ⚠️ Good process + wrong result → promising sample (possibly environmental noise)
• ❌ Poor process + wrong result → clearly erroneous sample

Traditional evaluation only sees outcomes and mislabels the second category as high quality. SkillCoach surfaces these hidden unreliable behaviors.

3. Self-Evolution: From Gold Skills to Battle-Tested Rubrics

Step 1: Skill-Dependent Task Filtering

Not all tasks are suitable for evaluating skill use. SkillCoach uses a three-layer filtering protocol:

1. No-skill success rate — if the agent completes the task without any skills, the task isn't skill-dependent; exclude it 2. Gold-skill gain rate — does using the correct skill significantly improve success rate vs. not using it? If not, exclude 3. Critical step coverage — does the task require executing critical steps from the skill? If not, exclude

These three filters ensure test tasks genuinely focus on skill-use capability.

Step 2: Generate Initial Rubrics from Gold Skills

From the filtered gold-standard skills, the framework automatically extracts:

• Skill names and descriptions
• Critical step checklists
• Dependency graphs
• Output verification requirements

These form the initial four-dimensional rubric rules.

Step 3: Verification-Gated Self-Evolution

Initial rubrics are imperfect, so SkillCoach iteratively improves them in three steps:

1. Evidence-based scoring

• Run the initial rubric rules on real interaction trajectories
• Record per-dimension scores and deduction points

2. Local arbitration correction

• Compare the outcome signal (external verifier) against the process score
• Investigate inconsistencies: low process score but correct result → check for "lucky success"
• High process score but wrong result → check for environmental noise
• Adjust rubric weights and thresholds based on inconsistent cases

3. Verification-gated updates

• New rubric rules must pass a consistency check on a validation set
• If new rules cause more misjudgments, roll back to the old rules
• Only rules that improve evaluation accuracy are kept

This self-evolution mechanism lets SkillCoach adapt to real enterprise-grade skill libraries — complex environments full of semantically similar distractor skills.

4. Dual Value: Evaluation + Training Enhancement

SkillCoach is not just an evaluation tool; it is also a quality filter for training data.

Process Supervision > Outcome Supervision

Traditional training-data filtering:

• Keeps only outcome-successful trajectories → mixes in many "lucky success" samples
• Discards outcome-failed trajectories → may throw out excellent-process samples that failed due to environmental noise

SkillCoach's process-supervised filtering:

• Prioritizes "good process + correct result" samples
• Identifies and removes "poor process + correct result" lucky successes
• Retains "good process + wrong result" promising samples (for failure analysis)
• Uses the four-dimensional scores as rich supervision signals

Experiments show that training data filtered by SkillCoach significantly improves agent skill-use capability compared with purely outcome-based filtering.

5. Why This Matters

1. From Black-Box Evaluation to White-Box Auditing

Current agent evaluation is mostly black-box: input a task, check the result. SkillCoach opens the process black box, letting developers see exactly how the agent performs at every stage of skill use.

2. Exposing the Hidden Risk of "Lucky Successes"

The most dangerous outcome is not an agent doing something wrong, but an agent reaching a correct result through a wrong process. If such "lucky successes" become training data, wrong patterns get amplified. SkillCoach is the first systematic method to identify this risk.

3. A Hard Requirement for Enterprise Skill Libraries

Real enterprise skill libraries are not clean — they contain overlaps, distractors, and skills that are semantically similar but functionally different. SkillCoach's self-evolution mechanism handles this complexity, where traditional evaluation methods largely fail.

4. Evaluation as Training

SkillCoach's rubrics are themselves high-quality process supervision signals. This means the evaluation framework can be directly converted into a training framework, forming a closed loop of "evaluate → find errors → improve training → re-evaluate."

6. Limitations and Outlook

1. Dependence on skill definition quality

SkillCoach's rubrics are generated from gold-standard skills. If skill descriptions are unclear or step definitions incomplete, the initial rubrics will be biased. Self-evolution can partially correct this, but only with enough interaction trajectories.

2. Computational overhead

Four-dimensional evaluation requires detailed step-level analysis of every trajectory, costing more than simple outcome verification. Real-time evaluation scenarios may need optimization.

3. Generalization to new skills

When brand-new skills are introduced, self-evolution requires collecting fresh trajectories and adjusting rubric rules. Making rubrics adapt faster to new skills remains an open problem.

4. Combining with LLM-as-Judge

SkillCoach's current rubric rules are structured (based on critical steps, dependencies, etc.). Could combining them with the flexibility of LLM-as-Judge handle more open-ended, semantic skill-use scenarios?

7. One-Sentence Summary

SkillCoach's core insight: the quality of an agent's skill use should not be defined by task success or failure, but by the correctness of the process. Through four-dimensional process auditing (selection, following, composition, reflection) and a verification-gated self-evolution mechanism, SkillCoach both catches hidden "lucky success" errors and filters higher-quality training data than outcome supervision — a key step from black-box scoring to white-box auditing in agent evaluation.

Reference

• Paper: SkillCoach: Self-Evolving Rubrics for Evaluating and Enhancing Agentic Skill-Use (arXiv:2607.01874)
• Authors: Jiayin Zhu, Kelong Mao, Yudong Guo, Dengbo He, Sulong Xu, Simiu Gu, Yutao Yue
• Core framework: Self-Evolving Rubrics
• Four-dimensional assessment: skill selection, skill following, skill composition, skill-grounded reflection
• Key techniques: skill-dependency task filtering protocol + verification-gated self-evolution
• Dual value: improved evaluation quality + training data filtering
• Key design: separation of process scores and outcome signals
• Experimental finding: evolved rubric rules simultaneously improve evaluation quality and training effectiveness
• Application scenario: enterprise-grade skill libraries (complex environments containing distractor skills)