Alizishaan Khatri
5 min
LLM-based code generation is increasingly used in mission-critical pipelines, but security defenses rely on post-hoc tools like static analyzers, fine-tuned classifiers, or external LLM judges. These methods analyze emitted code after the fact while ignoring the generating model's own internal computations. This paper investigates whether an LLM's hidden activations already carry a recoverable signal about code vulnerability when it reads code as context.
The author extracts last-prefill-token activations from four diverse LLMs spanning three families: Granite-4.1-8B, Qwen3.5-9B, Qwen3.6-27B, and Gemma-4-12B. Rather than performing a layerwise sweep, the methodology focuses on the final hidden layer's output vector. Six-layer MLP probes with dropout and GELU activations are trained on these frozen representations using a class-weighted binary cross-entropy loss. The probes are evaluated on four function-level C/C++ vulnerability detection benchmarks: Devign, Big-Vul, Draper VDISC, and PrimeVul.
The MLP probes achieve an average F1 score of 41.7% across all model and dataset pairs using only 13.4 to 16.0 million parameters, which amounts to under 0.2% of the base models' sizes. Performance varies heavily by dataset rather than model family. On the Devign benchmark, the best probe configuration achieves 68.8% F1, matching the published fine-tuned classifier state-of-the-art of 67.9% despite reading a general-purpose, frozen LLM.
However, performance drops on more challenging datasets. On Big-Vul, probes reach up to 59.5% F1, and on Draper VDISC and PrimeVul, scores decline further to approximately 30.5% and 17.6% F1 respectively. A breakdown of Big-Vul classifications across common Common Weakness Enumerations reveals uniformly high accuracy across the top five vulnerability types, with CWE-119 proving to be the most difficult category for all four models.
These results provide early evidence that a coding LLM's latent representation of arbitrary code contains meaningful vulnerability signals. From a systems perspective, internal probing offers massive computational advantages over external screening. External detectors require secondary models and multi-stage API round-trips, whereas a latent probe executes concurrently within the primary model's inference pass at a negligible parameter and latency cost.
This makes probe-based filtering particularly appealing for air-gapped, on-device, or tactical environments where sending generated code to an external security service introduces severe latency overheads or data exfiltration risks.
LLM-based code generation is now embedded in mission-critical pipelines, but defenses against vulnerable output remain post-hoc -- static analyzers, fine-tuned classifiers, or an LLM judge that screen completed code, ignoring the generating model's own internal state. We test a narrower, directly measurable question: when an LLM reads a piece of C/C++ code as context, do its hidden activations already carry a signal about that code's vulnerability status? We extract last prefill token activations from four LLMs (Granite-4.1-8B, Qwen3.5-9B, Qwen3.6-27B, Gemma-4-12B) across three model families and train MLP probes on these activations. We evaluate them on four function-level C/C++ benchmarks (Devign, Big-Vul, Draper VDISC, PrimeVul). Our probes achieve 41.7\% average F1 using 13.4--16.0M-parameter probes -- under 0.2\% of base-model size. On Devign, the best probe (Qwen3.5-9B, 68.8\% F1) matches the published fine-tuned-classifier SOTA (67.9\%) despite reading only a frozen, general-purpose LLM's activations; on the harder, more imbalanced benchmarks (Big-Vul, Draper VDISC, PrimeVul) probes trail SOTA substantially. This is early evidence that a coding LLM's own representation of arbitrary code is informative about that code's vulnerability status, motivating further work toward lightweight, model-native vulnerability screening.
Alex: That's where the picture gets more complicated. On PrimeVul — which is more imbalanced and generally considered noisier — the probes trail the state-of-the-art by a meaningful margin. The authors attribute this to dataset quality rather than a failure of the probing approach, but that's a hypothesis, not a demonstrated result. A careful referee would note that you can't cleanly separate "the probe lacks signal" from "the labels are unreliable" without a cleaner ground-truth set.
Sam: And there's a more fundamental issue here, isn't there? The whole pitch is generation-time gating, but the evaluation is on existing, human-labeled functions — not actual model completions.
Alex: Exactly. That's the critical constraint on how much weight these results can bear. Evaluating on corpus functions is a necessary proxy, but it doesn't directly test whether the probe generalizes to the model's own outputs, which may have different distributional properties. There's no cross-dataset generalization matrix, no adversarial robustness check — those are the controls that would tell you whether the probe is genuinely capturing vulnerability structure or overfitting to the statistical regularities of these particular benchmarks.
Sam: So the gap between "this works on labeled datasets" and "this works as an inline gate on model-generated code" is still open.
Alex: It is, and that's the honest characterization of where this sits. It's a workshop-scale study — the authors are transparent about that — and the contribution is establishing that the signal exists and is extractable cheaply. The harder questions about adversarial robustness, distribution shift to model completions, and whether the probe degrades on obfuscated or novel vulnerability patterns are left for follow-on work.
Sam: Still, if that signal is real, the downstream application is genuinely useful. A model that can flag its own vulnerable completions in real time, without routing code to an external service, would change how you'd architect a secure coding assistant.
Alex: That's the promise, and the efficiency story is what makes it plausible — a probe this small doesn't require a separate inference pass. The question is whether the next round of experiments, on actual model outputs and adversarial inputs, holds up the headline result. If it does, this becomes a meaningful building block. If the signal degrades under distribution shift, then what the probe is learning is more about benchmark structure than vulnerability semantics. Either way, it's a well-posed question and a clean experimental setup for asking it.
Sam: It's a good reminder that the interesting signal often lives inside the model, not just at the output layer.
Alex: Thanks for listening to ResearchPod.