Overview
Paper: SkillCoach: Self-Evolving Rubrics for Evaluating and Enhancing Agentic Skill-Use Authors: Jiayin Zhu, Kelong Mao, Yudong Guo, Dengbo He, Sulong Xu, Simiu Gu, Yutao Yue Link: https://arxiv.org/abs/2607.01874
TL;DR: Stop judging agents solely on task completion. SkillCoach audits the *process* of skill use across four dimensions, exposes 'lucky successes' hidden in current benchmarks, and doubles as a high-quality training-data filter.
---
The Blind Spot in Current Agent Evaluation
Most agent benchmarks rely on binary outcome metrics: task succeeded or failed. This works for simple settings but collapses in real skill-library environments because agents can reach correct results through four types of unreliable behavior:
1. Distractor skill selection — choosing a semantically similar but functionally wrong skill, then working around it 2. Skipped critical steps — bypassing required checks that happen to be unnecessary for the given input 3. Mis-composed workflows — wrong skill dependencies that don't break the specific case 4. Missing validation — skipping output verification that the external environment happens to forgive
All four patterns register as "success" under outcome-only evaluation, yet they encode unreliable behavior. Training on such trajectories bakes faulty skill-use patterns into the model.
---
Core Idea: Process Quality ≠ Outcome Success
SkillCoach reframes skill-use evaluation from outcome-oriented to process-oriented. The framework defines four trajectory-level meta-capabilities:
1. Skill Selection
- Penalizes missed gold-standard skills
- Penalizes selected distractor skills (semantically similar but functionally different)
- Uses an F1-style score balancing precision and recall
- Verifies coverage of required execution steps
- Penalizes skipped mandatory operations
- Validates cross-skill dependency handling
- Penalizes incorrect workflow chaining
- Checks whether the agent performed skill-mandated output validation
- Penalizes 'execute-and-exit' behavior with no verification
- Process score (SkillCoach's four dimensions)
- Outcome signal (external verifier: success/failure)
- ✅ Good process + correct outcome → high-quality sample
- ⚠️ Poor process + correct outcome → 'lucky success' (dangerous)
- ⚠️ Good process + wrong outcome → promising sample (possible environmental noise)
- ❌ Poor process + wrong outcome → clear error sample
- Skill names and descriptions
- Critical step lists
- Dependency graphs
- Output validation requirements
- Keeps outcome-successful trajectories (contaminated with 'lucky successes')
- Discards outcome-failed trajectories (may include strong processes masked by environment noise)
- Prioritizes 'good process + correct outcome'
- Removes 'lucky successes' that would amplify faulty patterns
- Retains 'good process + wrong outcome' samples for failure analysis
- Provides rich multi-dimensional supervision signals
- Paper: SkillCoach: Self-Evolving Rubrics for Evaluating and Enhancing Agentic Skill-Use (arXiv:2607.01874)
- Authors: Jiayin Zhu, Kelong Mao, Yudong Guo, Dengbo He, Sulong Xu, Simiu Gu, Yutao Yue
- Core framework: Self-Evolving Rubrics
- Four dimensions: Skill Selection, Skill Following, Skill Composition, Skill-Grounded Reflection
- Technical pillars: Skill-dependent task filtering protocol + validation-gated self-evolution
- Dual value: Evaluation quality uplift + training-data filtering
- Key design: Decoupling process score from outcome signal
- Experimental finding: Evolved rubrics simultaneously improve evaluation quality and downstream training
- Target scenario: Enterprise skill libraries with distractor skills
2. Skill Following
3. Skill Composition
4. Skill-Grounded Reflection
Key Design: Keep the External Verifier as an Independent Signal
SkillCoach runs in parallel with, not as a replacement for, the external verifier:
This yields a 2×2 quadrant:
Traditional evaluation conflates the first two rows. SkillCoach isolates the second — the most harmful one for training.
---
Self-Evolving Rubric Mechanism
Step 1: Skill-Dependent Task Filtering
Not every task meaningfully tests skill use. A three-layer filter excludes non-skill-dependent tasks:
1. No-skill success rate — if the agent succeeds without any skill, the task doesn't depend on skills 2. Gold-skill gain rate — does using the correct skill significantly raise success rate? 3. Critical-step coverage — does the task actually exercise the skill's key steps?
Step 2: Initial Rubric Generation from Gold Skills
From filtered gold-standard skills, the framework auto-extracts:
These seed the initial four-dimensional scoring rules.
Step 3: Validation-Gated Self-Evolution
Initial rubrics are imperfect. Three sub-steps drive iterative refinement:
1. Evidence-based scoring — run rules on real interaction trajectories, record per-dimension scores and deductions 2. Local arbitration correction — compare process scores against outcome signals; investigate mismatches to fix weights and thresholds 3. Validation gate — new rubrics must pass consistency checks on a held-out set; if they produce more misjudgments, revert to the prior version. Only accuracy-improving updates are retained
This mechanism is what enables adaptation to messy, real-world enterprise skill libraries full of overlapping and distractor skills.
---
Dual Value: Evaluation + Training Enhancement
SkillCoach doubles as a training-data quality filter.
Process Supervision > Outcome Supervision
Traditional filtering:
SkillCoach's process-supervised filtering:
Experiments report that data filtered by SkillCoach yields significantly stronger skill-use capabilities than outcome-only filtering.
---
Why It Matters
1. From black-box scoring to white-box auditing — developers see exactly where in the skill-use pipeline the agent underperforms 2. Exposes the 'lucky success' risk — the first systematic method to surface this hidden failure mode 3. Enterprise-skill-library ready — designed for noisy environments with overlapping and distractor skills 4. Evaluation-as-training — process scores are directly usable as training supervision, enabling a closed 'evaluate → find errors → improve training → re-evaluate' loop
---
Limitations and Open Questions
1. Rubric quality depends on skill definitions — unclear skill descriptions degrade the initial rubric; self-evolution partially compensates but requires sufficient trajectories 2. Computational overhead — step-level four-dimensional analysis is more expensive than simple outcome checks 3. Generalization to new skills — adding a novel skill requires new trajectories and rubric updates 4. LLM-as-Judge integration — current rubrics are structured; combining with LLM-as-Judge flexibility for more open-ended, semantic skill use is an open direction
---
One-Sentence Summary
Agent skill-use quality cannot be defined by task outcomes — it must be defined by process correctness. Through four-dimensional process auditing (selection, following, composition, reflection) and validation-gated self-evolution, SkillCoach detects hidden 'lucky successes' and produces higher-quality training data than outcome-only filtering, marking a key step from black-box scoring toward white-box auditing of AI agents.
---