VLoc Bench: A Benchmark Measuring Agentic Vulnerability Localization Across Real-World Code Repositories

Forum topic · 小凯 · 2026-09-16

Summary

Researchers introduce VLoc Bench, a benchmark for vulnerability localization: given a weakness class (CWE) and an unfamiliar repository, a language-model agent must identify the implementation files associated with that weakness. The benchmark contains 500 real-world vulnerabilities drawn from 290 repositories across six package ecosystems and 147 CWE categories. Each task pairs repository snapshots taken immediately before and after a security fix; on the vulnerable snapshot the agent receives only the CWE description and read-only terminal access and must return affected files, while on the patched snapshot it must decide whether the vulnerability has been removed. Evaluations covered 27 language models and 4 static analysis tools under a unified agentic interface. Results show repository-scale localization remains very hard: the best system achieves a file-level F1 of only 0.229, and 38.4% of tasks were not correctly localized by any evaluated model. Notably, stronger localization does not guarantee reliable post-fix behavior, as capable systems still report unfounded locations on patched codebases. The work establishes vulnerability localization as a distinct repository-level capability for studying security agents.

Overview

This post summarizes the paper "Vulnerability Localization Benchmark: Measuring Agentic Security Analysis" (arXiv: 2609.15939), authored by Aman Priyanshu, Supriti Vijay, Kimia Majd, Xuhong He, Fraser Burch, Takahiro Matsumoto, Jianliang He, Baturay Saglam, Arthur Goldblatt, Zhuoran Yang, and Amin Karbasi.

Field: Machine Learning
Published: 2026-09-14
arXiv: 2609.15939

Key Points

Task studied: Vulnerability localization — given a weakness class (CWE) and an unfamiliar repository, an agent must identify the implementation files associated with that weakness. This differs from prior cybersecurity evaluations focused on detecting, reproducing, or repairing vulnerabilities.
Benchmark design (VLoc Bench):
500 real-world vulnerabilities from 290 repositories
Six package ecosystems and 147 CWE categories
Each task pairs repository snapshots immediately before and after a security fix
Task setup: On the vulnerable snapshot, an agent receives only the CWE description and read-only terminal access, and must return the affected files. On the patched snapshot, it must determine whether the recorded vulnerability is no longer present.
Evaluation scope: 27 language models and 4 static analysis tools, all tested under a common agentic interface.

Findings

Repository-scale vulnerability localization remains difficult: the strongest system achieves a file-level F1 of only 0.229.
38.4% of tasks were not correctly localized by any evaluated model.
Stronger localization capability does not imply reliable behavior after fixes: systems that effectively identify vulnerable files may still report unfounded locations on patched repositories.

Significance

The results establish vulnerability localization as a distinct repository-level capability, providing scenarios for studying how security agents search for vulnerable code and when they should refrain from reporting.