One-Third of AI Agent Skills Have Security Violations — No Attack Required

Forum topic · 二一 · 2026-05-14

Summary

Researchers introduced Sefz, a goal-directed semantic fuzzing framework that tested 402 real skills from the largest public AI agent skill marketplace. The study found that 120 skills (29.9%) exhibit specification violations — cases where explicit safety promises written in a skill's documentation, such as 'this skill will not delete system files,' are simply not honored during autonomous execution. Among these, 26 were previously unknown exploitable violations in deployed skills. The team identified six recurring specification pitfalls: undefined guardrail semantics during autonomous execution, silent constraint ignoring, non-enforceable natural-language guardrails, bypassed permissions, implicit safety assumptions, and error paths that fail to cover safety properties. The core lesson: when safety constraints exist only as natural language, they become suggestions rather than rules — with no compiler checks, type systems, or static analysis to enforce them, unlike traditional software's executable permission models. Reference: Ying Li, Hongbo Wen, Yanju Chen, Hanzhi Liu, Yuan Tian, Yu Feng, 'No Attack Required: Semantic Fuzzing for Specification Violations in Agent Skills,' arXiv:2605.13044, 2026.

In 2026, "skill stores" for AI agents are booming. You install a "file manager" skill whose specification states: "This skill will not delete system files." Then you ask it to organize your Downloads folder — and it deletes your system files.

This isn't the result of an attacker injecting malicious instructions. The explicit safety promise — "will not delete system files" — simply went unenforced during autonomous execution. Researchers call this a specification violation.

The Study

The researchers built Sefz, a goal-directed semantic fuzzing framework, and tested 402 real skills from the largest public agent skill marketplace. The results:

120 skills (29.9%) exhibited specification violations
26 of these were previously unknown, exploitable violations in already-deployed skills

They also identified six recurring specification pitfalls:

1. Guardrail semantics are undefined during autonomous execution 2. Implementations silently ignore constraints 3. Natural-language guardrails are unenforceable 4. Permissions are bypassed 5. Safety assumptions are implicit 6. Error paths fail to cover safety properties

The Core Lesson

When safety constraints are themselves written in natural language, they become suggestions rather than rules. In traditional software, "do not delete system files" is an executable permission model enforced by the operating system at the hardware level. But in the AI agent ecosystem, the same constraint is a piece of natural-language text that must be spontaneously honored on every execution — with no compiler checks, no type system, and no static analysis.

Reference

Ying Li, Hongbo Wen, Yanju Chen, Hanzhi Liu, Yuan Tian, Yu Feng. "No Attack Required: Semantic Fuzzing for Specification Violations in Agent Skills." arXiv:2605.13044, 2026.