Cloudflare Open-Sources security-audit-skill: Turning AI Coding Agents into Adversarial Security Auditors

> Repository: cloudflare/security-audit-skill > Captured: 2026-09-16 | Stars at time: 1249 > License: MIT | Language: JavaScript

A Counter-Intuitive Design

Most AI security tools aim to "make the AI smarter at finding problems." Cloudflare's skill takes the opposite approach—make it harder for the AI to confirm a finding.

That sounds like friction, but consider: the biggest enemy of security auditing is not missed findings, it's false positives that force teams to spend days chasing non-existent vulnerabilities. False positives are costlier than false negatives because they trigger expensive verification workflows.

The skill's core principle is one sentence: "The agent that checks a finding is never the agent that found it."

It is separation of powers—the US constitutional principle of checks and balances—applied to AI security auditing. The finder cannot self-certify; the verifier cannot be the finder.

The Six-Stage Pipeline

The skill splits an audit into six stages, each run by independent agents:

1. Reconnaissance: Map the architecture, trust boundaries, and input surface into architecture.md and coverage-ledger.json. No vulnerability hunting here—just cartography. 2. Coverage-led hunting: Independent hunters are assigned from the ledger, each owning one attack-surface unit. Hunters log what they checked; a coverage critic checks for gaps. 3. Candidate validation: Each candidate finding goes to a fresh verifier agent whose job is to try to refute it—not "check if it's right" but "try to prove it wrong." 4. Structured output: Results go into findings.json, categorized as confirmed, needs_validation, or rejected, validated against report-schema.json. 5. Independent record verification: A fresh set of agents re-verifies source citations in the final results. If materials were swapped, an independent re-verification is done. 6. Target-neutral reporting: Reports depend only on static source-code facts, not on the target system's runtime state.

The key is that every step's agent is independent. The hunter doesn't know how the verifier will check; the verifier doesn't know how the hunter found it. This information isolation prevents confirmation bias—a hunter tends to believe it found a real bug, and if the same agent verifies it, it will almost certainly confirm its own finding.

"Multiple Runs Improve Coverage"

The README contains an understated but important line:

> In our test runs, a single run found roughly half of the vulnerabilities that repeated runs found in total.

A single run finds only about half of the total vulnerabilities. This isn't a flaw—it's an inherent property of LLM-based security auditing. LLM sampling is random; each run takes different paths and covers different code regions.

This matches penetration-testing experience: two different pentest teams auditing the same system often overlap on only 40–60% of findings. Security auditing is fundamentally a coverage problem, not an accuracy problem.

The skill's designers turned this into an advantage: multiple runs aren't a bug, they're a feature. Each run contributes to the coverage ledger, progressively approaching complete coverage. That's more honest than the illusion of "one run and trust it."

"Defense-in-Depth Gaps Are Not Vulnerabilities"

Another counter-intuitive principle:

> Defense-in-depth gaps are not vulnerabilities. If Layer A prevents the attack, the absence of Layer B is a hardening note.

This runs counter to most scanners, which flag "missing input validation" as a vulnerability even when parameterized queries already block SQL injection. The result: security reports full of "theoretically risky" items that exhaust teams.

Cloudflare's principle: severity requires impact, not checklist deviation. Likelihood × Impact, not "non-compliance."

Sandbox Requirements

The skill has strict runtime requirements:

> An OS-enforced sandbox for target-controlled builds, tests, processes, browsers, emulators, fuzzers, and fixtures. It must disable external networking, use a sanitized allowlisted environment, enforce resource limits, and allow writes only to assigned scratch paths.

Without a sandbox, the skill marks all findings as needs_validation instead of confirmed—it doesn't trust its own execution environment. Another "adversarial toward itself" design: if you can't guarantee the environment is safe, don't trust the results.

What Is This the Seed Of?

The README concludes:

> This is the skill that seeded Cloudflare's vulnerability discovery harness, described in Build your own vulnerability harness. The harness grew into a multi-stage, fleet-wide system; this skill is the single-repo starting point it evolved from.

This skill is the single-repo starting point of Cloudflare's internal vulnerability discovery system. The internal system has scaled to multi-stage, fleet-wide operation; what's open-sourced is its original form—a seed that can run standalone.

This "open-source the seed + internal forest" model is interesting. It's not "open-source a castrated version"—it's "open-source the seed that can grow into the forest." Anyone can start from the same seed and grow their own system.

How It Differs from Other AI Security Approaches

| Approach | Core idea | Problem | |----------|-----------|---------| | Static analysis (SAST) | Rule matching | High false positives, no semantic understanding | | Single-pass LLM audit | One-shot AI run | Low coverage, confirmation bias | | LLM + self-verification | AI checks itself | Confirmation bias persists | | security-audit-skill | Independent adversarial verification by separate agents | Requires multi-agent orchestration |

The key difference is between rows three and four. "Self-verification" sounds reasonable—let the finding agent re-check its work—but that's precisely a breeding ground for confirmation bias. An agent tends to trust its own judgment, especially after spending tokens reasoning about a vulnerability.

security-audit-skill's solution is information isolation: the verifier never sees the hunter's reasoning, only the candidate conclusion, and verifies it independently from scratch—like blind peer review in academic publishing, where reviewers can't relax standards based on the author's reputation.

A Larger Pattern

Behind this skill lies a bigger pattern: AI system reliability comes from structural design, not model capability.

A vanilla LLM doing security auditing won't perform well. But put the same LLM inside a well-designed multi-agent structure—hunter / verifier / critic each playing their role, information isolated—and performance improves dramatically.

It's the same as AlphaGo: a policy network alone isn't strong enough, but policy network + value network + MCTS together are. Structure > model.

security-audit-skill applies this to security auditing. It doesn't rely on a "smarter model" but on a "smarter structure." The model is swappable—GPT-4, Claude, Gemini all work—while the structure stays constant.

That's also why Cloudflare dares to open-source it: the competitive edge isn't the skill itself, but the know-how of "how to design multi-agent structures." The skill is the seed; the forest is internal.

---

Project: https://github.com/cloudflare/security-audit-skill Blog post: https://blog.cloudflare.com/build-your-own-vulnerability-harness License: MIT