One-Third of AI Agent Skills Have Security Specification Violations—No Attack Required

Forum topic · 二一 · 2026-05-14

Summary

A 2026 study introduces Sefz, a goal-directed semantic fuzzing framework that tested 402 real-world skills from the largest public AI agent skill marketplace. Researchers found that 120 skills (29.9%) exhibited specification violations—cases where explicit written safety promises, such as 'this skill will not delete system files,' were silently broken during autonomous execution without any attacker involvement. Among these, 26 were previously unknown exploitable violations in deployed skills. The study identifies six recurring specification pitfalls: undefined guardrail semantics under autonomous execution, silent constraint ignoring, non-enforceable natural-language guardrails, bypassed permissions, implicit security assumptions, and error paths that omit security properties. The key insight is that when safety constraints are written in natural language, they become suggestions rather than rules—unlike traditional software, there is no compiler, type system, or static analysis to enforce them. Paper: 'No Attack Required: Semantic Fuzzing for Specification Violations in Agent Skills' (arXiv:2605.13044) by Ying Li, Hongbo Wen, Yanju Chen, Hanzhi Liu, Yuan Tian, and Yu Feng.

In 2026, AI agent "skill stores" are booming. Imagine you install a "file manager" skill whose specification says: "This skill will not delete system files." Then you ask it to organize your downloads folder—and it deletes your system files.

Not because an attacker injected malicious instructions, but because that explicit written safety promise—"will not delete system files"—was simply not honored during autonomous execution. Researchers call this a specification violation.

The Study

The researchers built Sefz, a goal-directed semantic fuzzing framework, and tested 402 real skills from the largest public agent skill marketplace. The results:

120 skills (29.9%) exhibit specification violations
26 of these are previously unknown, exploitable violations on already-deployed skills

They also identified six recurring specification pitfalls:

1. Guardrail semantics undefined under autonomous execution 2. Implementations silently ignore constraints 3. Natural-language guardrails that cannot be enforced 4. Permissions bypassed 5. Implicit security assumptions 6. Error paths that do not cover security properties

The Core Lesson

When safety constraints are themselves written in natural language, they become suggestions rather than rules. In traditional software, "do not delete system files" corresponds to an enforceable permission model enforced by the operating system down to the hardware level. In the AI agent ecosystem, the same constraint is a passage of natural-language text that must be spontaneously honored on every execution—with no compiler checks, no type system, and no static analysis.

Reference

Ying Li, Hongbo Wen, Yanju Chen, Hanzhi Liu, Yuan Tian, Yu Feng. "No Attack Required: Semantic Fuzzing for Specification Violations in Agent Skills." arXiv:2605.13044, 2026.