Author-updated Summary
Verified author edit
LLM-based code generation is increasingly used in mission-critical pipelines, but security defenses rely on post-hoc tools like static analyzers, fine-tuned classifiers, or external LLM judges. These methods analyze emitted code after the fact while ignoring the generating model's own internal computations. This paper investigates whether an LLM's hidden activations already carry a recoverable signal about code vulnerability when it reads code as context.
The author extracts last-prefill-token activations from four diverse LLMs spanning three families: Granite-4.1-8B, Qwen3.5-9B, Qwen3.6-27B, and Gemma-4-12B. Rather than performing a layerwise sweep, the methodology focuses on the final hidden layer's output vector. Six-layer MLP probes with dropout and GELU activations are trained on these frozen representations using a class-weighted binary cross-entropy loss. The probes are evaluated on four function-level C/C++ vulnerability detection benchmarks: Devign, Big-Vul, Draper VDISC, and PrimeVul.
The MLP probes achieve an average F1 score of 41.7% across all model and dataset pairs using only 13.4 to 16.0 million parameters, which amounts to under 0.2% of the base models' sizes. Performance varies heavily by dataset rather than model family. On the Devign benchmark, the best probe configuration achieves 68.8% F1, matching the published fine-tuned classifier state-of-the-art of 67.9% despite reading a general-purpose, frozen LLM.
However, performance drops on more challenging datasets. On Big-Vul, probes reach up to 59.5% F1, and on Draper VDISC and PrimeVul, scores decline further to approximately 30.5% and 17.6% F1 respectively. A breakdown of Big-Vul classifications across common Common Weakness Enumerations reveals uniformly high accuracy across the top five vulnerability types, with CWE-119 proving to be the most difficult category for all four models.
Alex: Welcome to another episode of ResearchPod. Today we're looking at a study that asks a pointed question: does a large language model already "know" that a piece of code is vulnerable before it finishes generating it?
Sam: That's a different framing than I'd expect. Most security tooling treats the model as a black box and screens the output after the fact.
Alex: Right, and that's exactly the gap this paper is targeting. Static analyzers, external classifiers — they all run post-hoc, which adds latency and, if you're routing sensitive code to an external API, introduces its own attack surface. The authors' bet is that vulnerability semantics are already encoded in the model's internal activations, and that you can extract them with a lightweight probe without touching the model weights at all.
Sam: So they're reading the model's internal state rather than its output. What does that look like mechanically?
Alex: They take a frozen LLM and pull the hidden state at the last prefill token — the moment just before generation starts. That's a single vector representing everything the model has processed about the context so far. They then train a small MLP probe on those frozen activations to classify whether the code is vulnerable. The probe itself is under 0.2% the size of the base model, so it's cheap enough to run concurrently with generation rather than as a separate sequential step.
Sam: That's the efficiency argument. If the probe runs in parallel, you're getting a security gate for essentially no added latency. What does the performance look like?
Alex: The headline result is that this probe matches published state-of-the-art on Devign — a widely used vulnerability detection benchmark — without any fine-tuning on the downstream task. That's the load-bearing finding. A frozen-activation probe, trained only to read internal states, reaches the same F1 as classifiers that were explicitly trained for the job.
Sam: Which raises the obvious question: is it actually learning vulnerability semantics, or picking up on some surface-level confound — code length, function complexity, something like that?
Alex: The authors address this by breaking down accuracy across Common Weakness Enumeration types on the Big-Vul dataset. The idea is that if the probe were latching onto a superficial feature, you'd expect performance to cluster around a few CWE categories and collapse on others. Instead, accuracy is reasonably consistent across the most frequent vulnerability types — which is at least suggestive that the signal has some semantic grounding rather than being an artifact of one bug class.
These results provide early evidence that a coding LLM's latent representation of arbitrary code contains meaningful vulnerability signals. From a systems perspective, internal probing offers massive computational advantages over external screening. External detectors require secondary models and multi-stage API round-trips, whereas a latent probe executes concurrently within the primary model's inference pass at a negligible parameter and latency cost.
This makes probe-based filtering particularly appealing for air-gapped, on-device, or tactical environments where sending generated code to an external security service introduces severe latency overheads or data exfiltration risks.
AI-generated third-party summary by ResearchPod. Not official content or an endorsement by the paper authors or affiliated organizations.
Sam: Suggestive, but not conclusive. What does performance look like on harder datasets?
Alex: That's where the picture gets more complicated. On PrimeVul — which is more imbalanced and generally considered noisier — the probes trail the state-of-the-art by a meaningful margin. The authors attribute this to dataset quality rather than a failure of the probing approach, but that's a hypothesis, not a demonstrated result. A careful referee would note that you can't cleanly separate "the probe lacks signal" from "the labels are unreliable" without a cleaner ground-truth set.
Sam: And there's a more fundamental issue here, isn't there? The whole pitch is generation-time gating, but the evaluation is on existing, human-labeled functions — not actual model completions.
Alex: Exactly. That's the critical constraint on how much weight these results can bear. Evaluating on corpus functions is a necessary proxy, but it doesn't directly test whether the probe generalizes to the model's own outputs, which may have different distributional properties. There's no cross-dataset generalization matrix, no adversarial robustness check — those are the controls that would tell you whether the probe is genuinely capturing vulnerability structure or overfitting to the statistical regularities of these particular benchmarks.
Sam: So the gap between "this works on labeled datasets" and "this works as an inline gate on model-generated code" is still open.
Alex: It is, and that's the honest characterization of where this sits. It's a workshop-scale study — the authors are transparent about that — and the contribution is establishing that the signal exists and is extractable cheaply. The harder questions about adversarial robustness, distribution shift to model completions, and whether the probe degrades on obfuscated or novel vulnerability patterns are left for follow-on work.
Sam: Still, if that signal is real, the downstream application is genuinely useful. A model that can flag its own vulnerable completions in real time, without routing code to an external service, would change how you'd architect a secure coding assistant.
Alex: That's the promise, and the efficiency story is what makes it plausible — a probe this small doesn't require a separate inference pass. The question is whether the next round of experiments, on actual model outputs and adversarial inputs, holds up the headline result. If it does, this becomes a meaningful building block. If the signal degrades under distribution shift, then what the probe is learning is more about benchmark structure than vulnerability semantics. Either way, it's a well-posed question and a clean experimental setup for asking it.
Sam: It's a good reminder that the interesting signal often lives inside the model, not just at the output layer.
Alex: Thanks for listening to ResearchPod.