Grading AI Agent Skills: A Survey of Skill Evaluation and Evolution

Start with a report card: 86 task questions, 11 professional domains, 7 model configurations, and deterministic graders. The same agent, without skills, achieved an average pass rate of 33.9%; with a set of human-refined skills, 50.5%. That is roughly a 16-point average gain—but the distribution is extremely uneven: +51.9 points in the medical domain, only +4.5 in software engineering. And there is an easily overlooked number: on 16 of those tasks, skills actually made things worse.

This exam is called SkillsBench, posted to arXiv in February. It answers a question nobody had seriously quantified before: exactly how many points are skills worth.

In June, the Rutgers team organized this new territory into a survey, *Agent Skill Evaluation and Evolution* (arXiv 2606.11435), gathering scattered work into two maps: how skills evolve (four routes), and how skills are examined (six kinds of exams).

What Is a Skill

The survey cites a definition as a four-tuple: when to use it (conditions), how to do it (steps), when it's done (termination criteria), and how it plugs into other skills (reuse interfaces). In everyday language: a recipe—suitable ingredients, method, doneness standard, and whether it can be combined with other dishes into a full meal.

The difference from prompts lies in reusability and composability: a prompt is a one-off instruction, while a skill is an asset you can accumulate in a library. But once the library grows, new questions arise—how to manage it, how to write exams for it, how to fix it when it breaks. The survey addresses the latter three.

Four Routes of Skill Evolution

Execution feedback. The most intuitive route: run it once, see what went wrong (runtime errors, wrong outputs, unmet task requirements), record the error patterns, and rewrite the skill to prevent recurrence. It's the same move as a human reviewing their notes after a mistake.

Trajectory distillation. SPARK trusts only "evidence produced by execution in the environment," not unverified plans—extracting skills from successful trajectories. Trace2Skill is finer-grained: it extracts separate patches from successful and failed trajectories, then merges duplicate fixes into one conflict-free skill file.

Compression and augmentation. Once a library grows large, duplication and conflicts appear. SkillNet collects skills from GitHub repositories and office documents, builds a similarity graph, and uses the graph to decide whether two skills should be merged or reused. SkillX goes the opposite way: it deliberately targets tools nobody has covered and that repeatedly fail, synthesizing new tasks to fill gaps in the skill library.

Reinforcement learning. D2Skill has the cleverest design: run the same task twice—once with the retrieved skill, once without—and use the difference in success rates as the reward. Whether a skill helps becomes a directly trainable signal rather than a human's gut-feel evaluation.

The four routes are not mutually exclusive schools; they are more like four stages of a pipeline: run it (feedback), accumulate it (distillation), clean it up (compression), train it in (RL).

Six Kinds of Exams

Utility exams are the SkillsBench from the opening—+16 points on average, +51.9 in medical, only +4.5 in software engineering, with 16 tasks regressing.

Generation exams test the quality of skills a model produces. SkillLearnBench's finding is worth remembering: without external execution feedback, a model's self-generated skills accumulate errors and drift further off course the more it learns.

Routing exams test skill retrieval. SkillRouter accumulated 80,000 skills and 75 expert-validated queries, and measured a counterintuitive number: routing on only names and descriptions loses 31 to 44 percentage points of precision compared with routing on full text. Descriptions deceive the router; the body is what counts.

Safety exams are the most eye-catching. SkillGuardBench collected 581 skill packages, labeled benign, suspicious, or malicious, and catalogued three common hidden weapons: disguised rewrites, cloaked forwarding, and remotely fetched instructions. SKILL-INJECT goes further, planting malicious instructions inside legitimate skills to test whether agents will comply at runtime. The survey offers a cold verdict here: existing safety benchmarks all operate on "review before install, ignore after install"—no exam covers the long-term behavior of skills once in the library, especially while the library keeps evolving.

Software engineering exams. SWE-Skills-Bench pairs 49 public skills with 565 task instances. The survey's annotation is blunt: public repos pinned to fixed commits are reproducible, but proprietary workflows, legacy code, and continuously shifting engineering practices are all untested.

Real-environment exams. SkillForge uses 1,883 tickets and 3,737 tasks from five cloud vendors' technical support scenarios. WildClawBench drops the agent into a real personal-assistant environment with 60 tasks—a benchmark built by the survey's own authors, as they disclose.

Three Things the Survey Doesn't Solve

The future-directions section is restrained: most skill packages are purely textual procedural knowledge, incapable of handling multimodal scenarios involving screen viewing, web browsing, or robot manipulation—wherever visual state determines the next action, a recipe can't be written into a photograph. How to make good use of trajectory data, and how to turn skill safety from a gate check into continuous monitoring, both remain open questions.

Closing

The exams, the roadmap, and a 30-star repository (github.com/Cassie07/AgentSkill_Survey) are all there. To recompute SkillsBench's scores yourself, the original paper is arXiv 2602.12670.

The next time someone tells you "skills make agents stronger," you can first ask: how much did the average go up, which domains gained the most—and, which 16 tasks excepted.