Vulnerability Localization Benchmark: Measuring Agentic Security Analysis at Repository Scale

As language model agents increasingly perform security tasks, a critical gap remains: can these agents identify the specific files within an unfamiliar, full-scale software repository that contain a vulnerability? Existing benchmarks often focus on vulnerability detection (analyzing a pre-selected snippet) or downstream repair, leaving the repository-scale search capability poorly characterized. This paper introduces the Vulnerability Localization Benchmark (VLoc Bench) to evaluate this capability directly.

VLoc Bench consists of 500 real-world vulnerabilities across 290 repositories and six package ecosystems. Each task provides an agent with only a generic CWE description and read-only terminal access to a repository snapshot taken immediately before a security fix. The agent must identify the affected files (Phase A). Subsequently, the agent is presented with the patched repository and must correctly report that no vulnerability remains (Phase B). The authors evaluate 27 language models and four static-analysis tools, measuring performance via File F1 (for localization) and True Negative Rate (for verification).

Repository-scale vulnerability localization is difficult for current systems. The strongest model achieved only 0.229 File F1, and 38.4% of tasks were not correctly localized by any evaluated system. The authors find that repository structure—such as code concentration and directory depth—is a stronger predictor of difficulty than model size or architecture. Furthermore, there is a notable tension between localization and verification: models that are effective at finding vulnerable files often struggle to refrain from reporting false positives in patched repositories, highlighting that successful search does not imply reliable security judgment.

This work establishes vulnerability localization as a distinct, measurable security capability. By separating the search for vulnerable code from the downstream tasks of exploitation or repair, VLoc Bench provides a controlled environment to study how agents navigate complex codebases. The findings suggest that while model scale is important, effective security analysis requires specialized reasoning and reliable tool use, and that current agents often lack the calibrated restraint necessary for practical defensive assistance.