VLoc Bench: A Benchmark for Measuring Agentic Vulnerability Localization in Real-World Codebases

Forum topic · 小凯 · 2026-09-16

Summary

VLoc Bench is a new benchmark for evaluating whether language-model agents can localize vulnerabilities in unfamiliar software repositories, rather than merely detect or repair them. The benchmark contains 500 real-world vulnerabilities drawn from 290 repositories across six package ecosystems and 147 CWE categories, with each task pairing repository snapshots taken immediately before and after a security fix. Given only a CWE description and read-only terminal access, agents must identify the affected implementation files on the vulnerable snapshot, and judge whether the recorded vulnerability has been removed on the patched snapshot. Evaluations covered 27 language models and 4 static analysis tools under a common agentic interface. Results show repository-scale vulnerability localization remains difficult: the strongest system achieved a file-level F1 of only 0.229, and 38.4% of tasks were not correctly localized by any evaluated model. Notably, stronger localization does not guarantee reliable post-fix behavior, as systems that identify vulnerable files well may still report unsupported locations on patched codebases. The benchmark establishes vulnerability localization as a distinct repository-level capability for security agent research. (arXiv: 2609.15939)

Introduction

This post introduces a paper on VLoc Bench, a benchmark for vulnerability localization by language-model agents (arXiv: 2609.15939, published 2026-09-14).

Authors: Aman Priyanshu, Supriti Vijay, Kimia Majd, Xuhong He, Fraser Burch, Takahiro Matsumoto, Jianliang He, Baturay Saglam, Arthur Goldblatt, Zhuoran Yang, Amin Karbasi

Key Points

Task: Vulnerability localization — given a weakness class (CWE) and an unfamiliar repository, identify the implementation files associated with that weakness. Most prior cybersecurity evaluations only measure detection, reproduction, or repair, not localization.
Dataset: 500 real-world vulnerabilities from 290 repositories, spanning six package ecosystems and 147 CWE categories. Each task pairs repository snapshots immediately before and after a security fix.
Protocol: On the vulnerable snapshot, the agent receives only the CWE description and read-only terminal access, and must return the affected files. On the patched snapshot, it must judge whether the recorded vulnerability has been removed — testing restraint against false reporting.
Evaluation: 27 language models and 4 static analysis tools under a unified agentic interface.

Findings

Repository-scale vulnerability localization remains very difficult: the strongest system achieved a file-level F1 of only 0.229.
38.4% of tasks were not correctly localized by any evaluated model.
Stronger localization does not imply reliable post-fix behavior: systems that effectively identify vulnerable files may still report unsupported locations on patched repositories.

Conclusion

The authors establish vulnerability localization as a distinct repository-level capability, providing scenarios for studying how security agents search vulnerable code and when they should exercise restraint in reporting.

Paper: arXiv:2609.15939