Cloudflare security-audit-skill: A Single Run Finds Half the Vulnerabilities, But Repeated Runs Accumulate
> Research subject: cloudflare/security-audit-skill
> Snapshot date: 2026-10-20 | Stars that day: 1680
> License: MIT | Language: JavaScript
---
A Counter-Intuitive Promise
Most security tools sell the pitch of "run once, find everything." The README of Cloudflare's skill contains a remarkably honest statement:
> In our test runs, a single run found roughly half of the vulnerabilities that repeated runs found in total.
A single run finds only about half of the total vulnerabilities. This is not a bug—it's a feature.
It sounds like an excuse, but think about it: decades of penetration-testing industry experience have long shown that when two different teams audit the same system, the intersection of their results is often only 40–60%. Security auditing is fundamentally a coverage problem, not a "correctness" problem. Any tool claiming "one run gives full coverage" is either lying or has never run against a sufficiently large codebase.
security-audit-skill turns this property into a design principle: repeated runs accumulate. Each run reads the previous coverage-ledger.json, assigns new hunters to uncovered areas, and re-verifies sections of source code that have changed. Old findings are carried forward if the source is unchanged; if the source has changed, they are re-validated.
This is far more honest than the illusion of "one run and you can trust it."
From Skill to Harness
The README contains a subtle but key sentence:
> This is the skill that seeded Cloudflare's vulnerability discovery harness, described in Build your own vulnerability harness. The harness grew into a multi-stage, fleet-wide system; this skill is the single-repo starting point it evolved from.
This skill is not the entirety of Cloudflare's internal security system—it is its starting point. Cloudflare's full system has evolved into a multi-stage, fleet-wide architecture, but this open-source version retains the "single-repo starting point" form.
This evolution path is worth noting: first build a working skill, then let it grow into a system. Many teams start by trying to design a complete security platform, end up writing design docs for three months, and produce no working code. Cloudflare's path is the reverse: first solve the minimal problem of "auditing one repository," discover bottlenecks through real usage, then gradually expand into a fleet-wide system.
It's the same MVP logic as startups, but applied to internal tooling.
The Six-Stage Pipeline: Information Isolation Is the Core
The skill breaks a security audit into six stages:
1. Reconnaissance: map the architecture, trust boundaries, and input surfaces into architecture.md and coverage-ledger.json. This step finds no vulnerabilities—it only draws the map.
2. Coverage-led hunting: independent hunters are assigned from the ledger, each responsible for a single attack-surface unit. Hunters record what they checked, and a coverage critic checks for omissions.
3. Candidate validation: each candidate vulnerability goes to a brand-new verifier agent whose job is to try to refute it.
4. Structured output: results are written to findings.json, classified as confirmed, needs_validation, or rejected.
5. Independent record verification: the final results are handed to a fresh set of agents to verify source references.
6. Target-neutral reporting: reports depend only on static facts from source code, not on runtime state.
The core design principle is one sentence: "The agent that checks a finding is never the agent that found it."
This is the separation of powers from the U.S. Constitution, transplanted into AI security auditing. Legislators cannot enforce, enforcers cannot judge. A finder cannot self-certify, and a verifier cannot find. This information isolation prevents confirmation bias—hunters tend to believe they found real vulnerabilities, and if the same agent verifies its own work, it will almost certainly confirm its own findings.
Three Verdicts, Not Two
Most security tools output a binary: vulnerable or not. security-audit-skill gives three verdicts:
- confirmed: has a complete source-code trace and bounded observations. Can be fixed directly.
- needs_validation: has one precise unresolved fact, with no severity rating. Needs a human to look at it.
- rejected: falsified candidates. Recorded to avoid rediscovering them next time.
- A coding agent supporting tool calls and parallel sub-agents (Claude Code, Codex, etc.)
- Node.js (to run the zero-dependency validator)
- An OS-enforced sandbox (for target-controlled builds, tests, processes, etc.). The sandbox must disable external networking, use an allowlisted environment, enforce resource limits, and allow writes only to specified paths. Without these controls, the skill keeps leads as
needs_validationrather than executing target code.
The needs_validation middle state is important. Most tools either report "confirmed vulnerability" or say nothing, but in the reality of security auditing, a large fraction of candidates are "look like vulnerabilities but can't be confirmed"—perhaps because the code path is unreachable, perhaps because input validation happens elsewhere, perhaps because runtime configuration blocks it. Reporting them as "confirmed" produces false positives; not reporting them misses real problems.
needs_validation makes this gray zone explicit. It doesn't force the agent into a binary judgment—it says "I found something, but a human needs to confirm it." This is far more useful than reports flooded with false positives.
"Defense-in-Depth Gaps Are Not Vulnerabilities"
The skill has an easily-missed design principle:
> Defense-in-depth gaps are not vulnerabilities. If Layer A prevents the attack, the absence of Layer B is a hardening note.
If Layer A already blocks the attack, the absence of Layer B is just a "hardening note," not a vulnerability.
This principle solves a problem that has long plagued security auditing: many tools report "missing best practices" as "vulnerabilities." For example, a system may already validate input at the gateway layer but not re-validate at the application layer—many tools would report "missing input validation at the application layer, high severity." In reality the gateway already blocks it, and application-layer validation is defense-in-depth, not a requirement.
Separating defense-in-depth gaps from vulnerabilities makes report priorities clearer. When developers see "high severity" they know it really needs fixing; when they see "hardening note" they know it's a nice-to-have.
What Counts as "Confirmed"
The README's definition of confirmed is strict:
> Only confirm established boundary failures. Keep a source-grounded blocked lead as needs_validation with its exact unresolved fact.
Only "established boundary failures" can be confirmed. If a lead is merely "blocked at the source level" (e.g., the code path looks reachable but is unreachable at runtime), it must remain at needs_validation, carrying the precise fact that remains unresolved.
This definition is much stricter than most tools. Most tools' "confirmed" just means "the agent thinks it's a vulnerability," but an agent's judgment may rest on incomplete code reading, wrong path assumptions, or ignoring some runtime check. security-audit-skill requires confirmed findings to have a complete source trace—from input to impact, with a source citation at every step.
Applicable Scenarios
This skill is not a universal solution. The README explicitly lists requirements:
needs_validation rather than risking the execution of potentially harmful code. This is respect for the principle that "a security tool must not itself become an attack surface."Lessons from the Design Philosophy
The design philosophy of security-audit-skill can be summarized in three points:
1. Information isolation > capability enhancement. Rather than making agents smarter at finding problems, make different agents check each other. Finders cannot verify; verifiers cannot find. 2. Coverage > correctness. The correctness of a single run doesn't matter; the accumulating coverage of repeated runs does. Admitting the limits of a single run is more honest than pretending one pass suffices. 3. Strict definitions > loose definitions. Confirmed requires a complete source trace, defense-in-depth gaps don't count as vulnerabilities, and needs_validation is an explicit middle state. Strict definitions make reports credible; loose definitions turn reports into noise.
These three lessons apply beyond security auditing. Any scenario where AI agents must make "judgments"—code review, test generation, documentation review—can borrow them: let different agents check each other, admit the limits of a single run, and strictly define output categories.
---
GitHub: https://github.com/cloudflare/security-audit-skill License: MIT Cloudflare blog: Build your own vulnerability harness