Cloudflare Open-Sources security-audit-skill: Turning AI Coding Agents into Adversarial Security Auditors
> Project: cloudflare/security-audit-skill
> Fetched: 2026-09-16 ｜ Stars: 1,249
> License: MIT ｜ Language: JavaScript
A Counter-Intuitive Design
Most AI security tools are designed to "make the AI smarter at finding issues." Cloudflare's skill takes the opposite approach—making it harder for the AI to confirm a finding.
It sounds like adding friction, but consider: the biggest enemy of security auditing isn't "missing a bug," it's "finding a false positive and then spending three days validating a vulnerability that doesn't exist." False positives cost more human effort than false negatives because they trigger expensive verification workflows.
The core design principle is a single sentence: "The agent that checks a finding is never the agent that found it."
This is the US Constitution's separation of powers—checks and balances among legislative, judicial, and executive branches—applied to AI security auditing. The finder cannot self-validate; the verifier cannot find.
The Six-Stage Pipeline
The skill splits a security audit into six stages, each executed by independent agents:
1. Reconnaissance: map the architecture, trust boundaries, and input surfaces into architecture.md and coverage-ledger.json. This step finds no vulnerabilities—it only draws the map.
2. Coverage-led hunting: independent hunters are assigned from the ledger, each responsible for one attack-surface unit. Hunters record what they checked; a coverage critic checks for gaps.
3. Candidate validation: each candidate vulnerability goes to a fresh verifier agent whose job is to try to refute it. Not "verify it's correct"—"try to prove it wrong."
4. Structured output: results are written to findings.json as confirmed, needs_validation, or rejected, validated against report-schema.json.
5. Independent record verification: the final results go to a fresh set of agents that verify source citations. If materials were swapped, an independent verification runs again.
6. Target-neutral reporting: the report relies only on static facts from source code, not the target's runtime state.
The key is that every agent in each step is independent. Hunters don't know how the verifier will check; verifiers don't know how the hunter found things. This information isolation prevents confirmation bias—hunters tend to believe they found real vulnerabilities, and if the same agent validates, it will almost certainly confirm its own finding.
"Repeated Runs Improve Coverage"
The README contains an easily overlooked but important sentence:
> In our test runs, a single run found roughly half of the vulnerabilities that repeated runs found in total.
A single run finds only about half the total vulnerabilities. This isn't a flaw—it's an inherent property of all LLM-based security auditing. LLM sampling is random; each run takes a different path and covers different code regions.
This matches penetration testing experience: two different pentest teams auditing the same system often have only 40–60% overlap in findings. Security auditing is fundamentally a coverage problem, not an "accuracy" problem.
The designers turned this property into an advantage: multiple runs are not a bug, they're a feature. Each run adds to the coverage ledger, progressively approaching complete coverage. That's more honest than the illusion of "run once and trust it."
"Defense-in-Depth Gaps Are Not Vulnerabilities"
Another counter-intuitive design principle:
> Defense-in-depth gaps are not vulnerabilities. If Layer A prevents the attack, the absence of Layer B is a hardening note.
In other words: if Layer A already blocks the attack, the absence of Layer B is just a "hardening suggestion," not a vulnerability.
This runs opposite to many security scanners. Most scanners flag "missing input validation" as a vulnerability even when parameterized queries already prevent SQL injection. The result is security reports full of "theoretically risky" items that exhaust teams.
Cloudflare's principle: severity requires impact, not deviation from a checklist. Likelihood × Impact, not "non-compliance."
Sandbox Requirements
The skill has strict environment requirements:
> An OS-enforced sandbox for target-controlled builds, tests, processes, browsers, emulators, fuzzers, and fixtures. It must disable external networking, use a sanitized allowlisted environment, enforce resource limits, and allow writes only to assigned scratch paths.
Without a sandbox, the skill marks all findings as needs_validation instead of confirmed—it doesn't trust its own execution environment. Another "adversarial toward itself" design: if you can't guarantee a safe environment, don't trust the results.
A Seed for What?
The README ends with:
> This is the skill that seeded Cloudflare's vulnerability discovery harness, described in Build your own vulnerability harness. The harness grew into a multi-stage, fleet-wide system; this skill is the single-repo starting point it evolved from.
This skill is the single-repo starting point of Cloudflare's internal vulnerability discovery system. The internal system has scaled to multi-stage, fleet-wide levels, but what's open-sourced is its original form—a seed that can run standalone.
This "open-source the seed + keep the forest internal" model is interesting. It's not "open-sourcing a crippled version"—it's "open-sourcing the seed that can grow into a forest." Anyone can start from the same seed and grow their own system.
How It Differs from Other AI Security Approaches
| Approach | Core Idea | Problem | |------|---------|--------| | Static analysis (SAST) | Rule matching | High false positives, no semantic understanding | | Single-run LLM audit | Run the AI once | Low coverage, confirmation bias | | LLM + self-verification | Let AI check itself | Confirmation bias persists | | security-audit-skill | Independent agent adversarial validation | Requires multi-agent orchestration |
The key difference lies between rows three and four. "Self-verification" sounds reasonable—have the agent that found the issue check it again—but that's precisely the breeding ground for confirmation bias. Agents tend to trust their own judgment, especially after spending tokens reasoning about a vulnerability.
security-audit-skill's solution is information isolation: the verifier doesn't see the hunter's reasoning, only the candidate conclusion, and validates from scratch. It's like blind peer review of academic papers—reviewers don't know the author, so they don't relax standards based on reputation.
A Larger Pattern
Behind this skill lies a bigger pattern: the reliability of AI systems comes from structural design, not model capability.
A vanilla LLM doing security auditing directly won't perform well. But put the same LLM into a carefully designed multi-agent structure—hunter / verifier / critic each with their role, information isolated between them—and performance improves dramatically.
It's the same principle as AlphaGo: a policy network alone isn't strong enough, but policy network + value network + MCTS together are. Structure > model.
security-audit-skill applies this to security auditing. It doesn't depend on a "smarter model"—it depends on a "smarter structure." The model can be swapped—GPT-4, Claude, Gemini all work—but the structure stays.
That's also why Cloudflare dares to open-source it: the competitive edge isn't in the skill itself, but in the know-how of "how to design multi-agent structures." The skill is the seed; the forest is internal.
---
Project: https://github.com/cloudflare/security-audit-skill Blog post: https://blog.cloudflare.com/build-your-own-vulnerability-harness License: MIT