HoF-Bench: Rediscovering Real AI-Discovered CVEs Without Frontier Models

HoF-Bench is a new benchmark designed to evaluate how effectively LLM-based analyzers can rediscover known, real-world vulnerabilities. Unlike synthetic benchmarks, HoF-Bench uses 95 public CVEs across eight mature open-source repositories (such as OpenSSL and curl) pinned at the exact vulnerable commit. The study employs a "source-conditioned" task, where the scanner is provided with the relevant target files but no information about the CVE identifier, description, or fix. The researchers tested ten non-frontier, open-weight, and proprietary "flash"-tier models within a fixed, multi-stage scanning scaffold to measure recall and reliability.

• Rediscovery Performance: The best-performing model (GPT-5.6 luna) successfully rediscovered 65 out of 95 CVEs (68%) using a four-pass scanning protocol.
• Diversity vs. Repetition: The study found that model diversity is highly effective; combining single passes from four different models yields better coverage than four repeated passes of a single model. This suggests that different models have non-overlapping blind spots.
• Language Sensitivity: Vulnerability detection difficulty is strongly structured by programming language. While JavaScript and PHP vulnerabilities were frequently found, C-based infrastructure code proved significantly harder, with 9 of the 11 CVEs missed by every model occurring in C.
• Pipeline Efficacy: Increasing the number of scanning passes consistently improved recall. Conversely, adding "generated context" (briefings on code flow) and increasing triage depth provided only marginal gains in recall while significantly increasing the volume of candidate findings, which would increase the burden on human reviewers.

This research shifts the focus of vulnerability scanning evaluation from synthetic snippets to real-world, complex software environments. By providing a standardized, low-barrier test bed, HoF-Bench allows developers to benchmark their own scanners against a set of high-stakes, AI-discovered bugs. The findings suggest that organizations can achieve high recall by deploying a portfolio of smaller, cost-effective models rather than relying on a single, expensive frontier model, provided they use a robust, multi-pass scanning and triage scaffold.