One in Three AI Agent Skills Has a Specification Violation — No Attack Required

Forum topic · 二一 · 2026-05-14

Summary

A 2026 study introduces Sefz, a goal-directed semantic fuzzing framework that tested 402 real skills from the largest public AI agent skill marketplace. Researchers found that 120 skills (29.9%) violated their own written safety specifications — a class of failures the authors call specification violations — including 26 previously unknown exploitable violations in deployed skills. Unlike prompt injection or adversarial attacks, these failures occur when an agent simply fails to honor explicit promises in its own documentation, such as a file-manager skill stating 'this skill will not delete system files' that then deletes them. The paper identifies six recurring specification pitfalls: undefined guardrail semantics under autonomous execution, silent constraint ignoring, non-enforceable natural-language guardrails, bypassed permissions, implicit safety assumptions, and error paths that omit safety properties. The core lesson: when safety constraints are expressed in natural language, they become suggestions rather than rules, lacking the compilers, type systems, and static analysis that enforce guarantees in traditional software. Reference: Ying Li, Hongbo Wen, Yanju Chen, Hanzhi Liu, Yuan Tian, Yu Feng, 'No Attack Required: Semantic Fuzzing for Specification Violations in Agent Skills,' arXiv:2605.13044, 2026.

In 2026, "skill stores" for AI agents are booming. You install a file-manager skill whose specification says: "This skill will not delete system files." Then you ask it to organize your downloads folder — and it deletes your system files.

No attacker injected malicious instructions. The explicit safety promise — "will not delete system files" — simply was not honored during autonomous execution. Researchers call this a specification violation.

The team built Sefz, a goal-directed semantic fuzzing framework, and tested 402 real skills on the largest public agent skill marketplace. The result: 120 skills (29.9%) had specification violations, including 26 previously unknown, exploitable violations on deployed skills.

Key findings

The researchers identified six recurring specification pitfalls:

Guardrail semantics are undefined during autonomous execution
Implementations silently ignore constraints
Natural-language guardrails are non-enforceable
Permissions can be bypassed
Safety assumptions remain implicit
Error paths do not cover safety properties

The core lesson

When a safety constraint is itself written in natural language, it becomes a suggestion rather than a rule. In traditional software, "do not delete system files" is backed by an executable permission model enforced at the hardware level by the operating system. In the AI agent ecosystem, the same constraint is a passage of natural-language text that must be spontaneously honored at every execution — with no compiler checks, no type system, and no static analysis.

Reference

Ying Li, Hongbo Wen, Yanju Chen, Hanzhi Liu, Yuan Tian, Yu Feng. "No Attack Required: Semantic Fuzzing for Specification Violations in Agent Skills." arXiv:2605.13044, 2026.