VEX-BENCH: Benchmarking LLM Agents for Assessing Exploitability of Software Supply Chain Vulnerabilities

Software supply chain security is a major concern, yet existing automated tools like GitHub Dependabot often generate excessive false alerts by flagging vulnerable dependencies without verifying whether they are actually exploitable in a downstream project. This paper introduces VEX-BENCH, the first benchmark designed to evaluate how effectively LLM agents can perform end-to-end exploitability assessments. Unlike previous cybersecurity benchmarks that focus on zero-day discovery, VEX-BENCH requires agents to reason across repositories to determine if a known upstream vulnerability is reachable and exploitable within a specific downstream codebase.

The benchmark consists of 75 real-world cases mined from GitHub, covering projects in Python, Java, and Go. Each case pairs a target codebase with a known CVE. Agents are tasked with producing a binary vulnerability status (affected or not affected) and, if not affected, a fine-grained justification (e.g., code not reachable, code not present, requires configuration, or requires environment). The dataset was manually annotated by security experts to ensure high-quality ground truth. The authors evaluated nine different model-harness configurations within isolated Docker environments to prevent benchmark leakage and ensure reproducibility.

While top-performing models like GPT-5.5 and Claude Opus 4.6 achieve approximately 80% F1 scores on binary vulnerability-status classification, their performance drops notably when tasked with identifying the specific reason for non-exploitability. Only GPT-5.5 surpassed a 70% macro-F1 score on fine-grained justification classification. The study reveals that traditional software composition analysis (SCA) tools, even those using reachability analysis, fail to match the performance of LLM agents, yet even the best agents struggle to maintain high accuracy as repository size increases. The results suggest that while LLMs are promising for triage, they are not yet fully reliable for the nuanced reasoning required in professional security workflows.