Patch2Vuln: Letting LLM Agents 'Read' Vulnerabilities from Binary Patches — UCL's 25-Case Empirical Study
Imagine a security analyst sitting before two monitors: on the left, yesterday's "healthy" binary; on the right, today's freshly pushed "fixed" version. Between them lies rows of hex and Ghidra's decompiled output. The analyst's task is not to write an exploit, but to answer a seemingly simple yet extremely hard question — what vulnerability did this patch actually fix?
That is the core setup of Patch2Vuln. Authors Isaac David and Arthur Gervais from University College London formalize this as vulnerability reconstruction from binary patch pairs. The agent sees no CVE pages, no security advisories, no source patches, and cannot go online. Its only tools are local binary evidence: ELF metadata, symbol tables, strings, Ghidra decompilation diffs, and limited local old/new behavioral validation.
1. Why Is This Problem Worth Studying?
Security patches create a peculiar time window: shortly after a patch is released, both attackers and defenders can see the difference between the "vulnerable old version" and the "fixed new version." Prior work — such as Brumley et al.'s patch-based exploit generation — proved that in some cases, comparing vulnerable and patched versions suffices to synthesize a working exploit. Systems like BinXray leverage this difference to build vulnerability signatures for large-scale one-day vulnerability identification.
But Patch2Vuln asks a different question: can an offline LLM agent reconstruct the security meaning of a vulnerability from local binary evidence alone? Not generating exploits, but locating security-relevant changed code regions, inferring root-cause categories (bounds check, integer overflow, null dereference, etc.), identifying input media, and proposing verifiable behavioral hypotheses.
> This framing matters. The paper explicitly states that a crashing PoC is a strong validation signal but not the primary metric. Many distribution patches add hard-to-trigger high-threshold integer limits, parser state guards, or defense-in-depth checks — these still deserve to be understood even if they aren't easily fuzzed.
2. System Architecture: A Candidate-Centered Three-Stage Audit
Patch2Vuln's core is not a new binary diff analyzer but an agentic harness that converts existing tooling (Ghidra + Ghidriff) output into structured, vulnerability-level explanations.
The pipeline has four deterministic stages and three agent stages:
| Stage | Type | Function |
|-------|------|----------|
| Extract old/new packages | Deterministic | dpkg-deb -x extraction, ELF metadata collection |
| Ghidra/Ghidriff diff analysis | Deterministic | Changed function list and decompiled snippets |
| Candidate ranking | Deterministic | Re-rank changed functions by memory-safety features (new comparisons against INT_MAX/SIZE_MAX, file length checks, memcpy argument changes, etc.) |
| Build candidate dossiers | Deterministic | For each ranked function: binary paths, ranking features, diff metadata, nearby strings, decompiled snippets, local call context |
| Preliminary audit | Agent | Infer vulnerability category, security-relevant functions, validation hypotheses from static binary evidence |
| Validation plan | Agent | Select safe local action templates (tcpdump_pcap, expat_xmlwf, libarchive_archive, etc.), harness renders bounded inputs |
| Final audit | Agent | Revise conclusions, confidence, and candidate ranking based on observed behavioral differences |
> Key design decision: prompt construction is structure-aware, each candidate has a fixed budget, candidate ranking order is preserved, and omitted counts are recorded. This avoids a common failure mode — evidence truncated mid-object, where the model sees neither the complete function dossier nor knows what was omitted.
3. Benchmark: 25 Ubuntu .deb Package Pairs
The evaluation covers 25 targets: 20 security update pairs + 5 negative controls. Targets span parser libraries (Expat, libxml2, SQLite, zlib), network/protocol (tcpdump, curl, OpenSSL, GnuTLS), archive/media (libarchive, TIFF, JPEG, WebP, Poppler), and noisy patch clusters (Ubuntu often bundles multiple CVE fixes into one release).
| Metric | Result | |--------|--------| | Security targets where the agent located the ground truth | 10 / 20 | | Security targets with correct root-cause classification | 11 / 20 | | Negative controls correctly rejected (unknown) | 5 / 5 | | Crashes / timeouts / sanitizer findings produced | 0 | | Behavioral differences produced (non-exploit) | 2 (both tcpdump) |
The failure analysis (Figure 2) is refreshingly honest:
- 6 rank/diff misses: the manually labeled ground-truth function never entered the candidate set — a binary diff coverage problem, not a model reasoning failure
- 1 context-export miss: the function was a candidate but Ghidra context export failed
- 3 model/validation misses: candidates reached the prompt, but the agent's reasoning or validation failed
4. Representative Cases
tcpdump bionic (success): The manually labeled filter-file path ranked 6th, was included in the prompt, selected for validation, and named in the final report. Validation used an oversized sparse filter file to produce a diagnostic difference — the old version reported a short-read mismatch and a negative expected size, while the new version rejected it as too large. This is the correct behavioral difference for CVE-2018-16301, not a crash.
tcpdump xenial (interesting failure): A large version jump with 496 changed functions; the ground-truth function never appeared in candidates. But bounded local search still found a 113-byte UDP pcap behavioral difference in the BFD packet parser — the old version printed Poll, Reserved, Reserved, Reserved while the new version printed Poll, Authentication Present, Reserved, Reserved. A real parser behavior change, just not the sealed-oracle-labeled function.
Expat (static reconstruction success, no trigger): The candidate ranked 1st, correctly classified as integer_overflow, describing size arithmetic and allocation/copy hardening. 41 bounded probes produced no old/new difference. The final report handled this correctly: static evidence supports integer overflow hardening, but actual triggerability was not demonstrated.
libarchive (conservative classification, correct): The agent located the true function family but ultimately classified it as unknown, because bounded validation failed to reproduce a behavioral difference and the patch cluster spanned multiple archive formats. 36 probes showed no difference. This is the correct behavior for a defensive audit agent — not erasing strong static evidence because bounded testing failed, but not overstating memory corruption either.
Negative controls (hallucination resistance): All 5 negative controls returned unknown. Two were byte-identical controls (tcpdump, Expat); three contained real changed functions but non-security fixes (WebP with 727, OpenJPEG with 303, zlib with 20). The agent was not tempted by the "a diff exists, so there must be a vulnerability" bias.
5. Three Key Honest Conclusions
Patch2Vuln's conclusions are not "we solved automated vulnerability analysis" but three precise honest assessments:
1. Agentic vulnerability reconstruction from binary patches is viable — not through automatic exploit generation, but by converting raw binary diffs into structured hypotheses.
2. The biggest bottleneck is not model reasoning but binary diff coverage and candidate reachability tracking — 6/10 failures happened before evidence reached the model.
3. Bounded local validation is a necessary but difficult complement — only 2 behavioral differences were found, no crashes, but those two differences already proved the changed parser path is reachable.
> The paper's ethics statement is also well placed: this work is inherently dual-use, but the system is explicitly scoped to understanding and auditing — the agent has no security advisories or network access during reconstruction, validation runs only on local old/new binaries in Docker, and reports produce vulnerability explanations and diagnostic differences, not exploit payloads.
Final Thoughts
Patch2Vuln suggests a broader observation: the most valuable application of LLM agents in security may not be replacing human hackers to write exploits, but converting low-signal-to-noise raw analysis output (hundreds of changed functions, synthetic function names, noisy decompiled snippets) into human-readable security narratives. In this sense, Patch2Vuln's 50% localization rate and 55% correct classification rate are not "not good enough" — they are an honest baseline that precisely tells us where current systems' boundaries lie and where to invest next (adjacent-version micro-diffs, better binary diff coverage, candidate reachability tracking).
That is far more valuable than papers that report only accuracy numbers without analyzing failure modes.
---
Paper Metadata
| Item | Detail | |------|--------| | Title | Patch2Vuln: Agentic Reconstruction of Vulnerabilities from Linux Distribution Binary Patches | | Authors | Isaac David, Arthur Gervais | | Institution | University College London | | arXiv ID | 2605.06601 | | Published | 2026-05-07 | | Link | https://arxiv.org/abs/2605.06601 | | Core contribution | Task definition and pipeline for agentic vulnerability reconstruction from Linux distribution binary patches; three-stage audit architecture; 25-case Ubuntu .deb benchmark | | Key results | 10/20 security targets located, 11/20 correctly classified by root cause; all 5 negative controls correctly rejected; 2 behavioral differences (non-exploit) |