LAST SECURE CODE BENCHMARK: IS THE CODE YOU VIBE SECURE?: Research Paper Podcast

Sam: So what separates a pass from a fail there?

Alex: One choice the specification leaves open: how to launch the process. Four of the six models passed the arguments as a list, so no shell ever interprets the paths. Claude Fable 5 even noted unprompted that this stops injection. The other two built a command string for the shell. Same specification, same functional result, and the injection stayed open.

Sam: So functionally they're indistinguishable. Only the attack test sees the difference.

Alex: That's exactly the gap the benchmark is built to expose.

Sam: Then give me the headline numbers, and what they're compared against.

Alex: Five of the six models passed the functional tests on about nine in ten tasks. But the best, Claude Fable 5, was both functional and secure on about 25 percent. The others ranged down to about 16 percent. The difference is almost all one outcome: code that works and keeps the vulnerability.

Sam: Does a model at least handle a given vulnerability consistently? If it secures the function version, does it secure the file and repository versions too?

Alex: Rarely. The best model secured all three versions of a scenario on only 12 percent of scenarios. So securing it once is no evidence the model understood it.

Sam: I'd guess bigger tasks are harder to secure. More code, more places to slip.

Alex: The document shows something different. Functional completion falls as scope grows. But functional-and-secure is flat or rising from function to repository for every model. The authors offer a possible reason: filling a hole inside existing code invites the agent to reproduce the original vulnerable design. Building a module from scratch, it picks the checks itself.

Sam: Which kinds of weaknesses do the agents get right, and which do they miss?

Alex: Memory safety is secured most often, roughly a third of attempts pooled across models. Those defenses are local, like a bounds check, and careful code includes them anyway. At the bottom are weaknesses whose defense is a policy nobody stated. Two are almost never secured. One is information exposure. The other is server-side request forgery, tricking a server into fetching things it shouldn't.

Sam: Is there a case that shows the policy problem clearly?

Alex: A tool called syft prints its configuration, which includes an attestation password. The agent had to rewrite a small config struct. Every model copied the tagging style of the neighboring fields, so the password got printed with everything else. No trajectory mentioned that the value was a password. The authors' point is that the check is easy to write but hard to know you need.

Sam: Surely just telling the agent to be secure fixes a lot of this.

Alex: Some of it. They added one sentence asking for secure code and reran four models. For three of them, functional-and-secure rose by roughly 15 to 21 points, with functional pass rates barely moving. Most tasks still stayed insecure, though. Information exposure and request forgery stayed near the bottom even with the reminder.

Sam: What about effort? Would more turns, more checking, buy more security?

Alex: Not in this data. The model using the fewest turns secured the most tasks. Others spent about twice as many turns and secured fewer. In one case, an agent wrote its own smoke test and grew it to thirty-three checks. It caught a corrupt-archive edge case. But every check used well-formed input. Its YAML parser, for reading uploaded config files, would build any class an attacker named. Those were the same hidden attacks two other models blocked with a safe parser.

Sam: Different models fail differently, then. Does pooling them help?

Alex: Pooled, at least one model secured about 44 percent of tasks, against 25 for the best alone. But the authors stress that union is picked in hindsight, so it's not a strategy you could run in advance. And more than half the tasks, 56 percent, were secured by no model at all.

Sam: What should make me cautious about these numbers?

Alex: A few things. Each task is attempted once, deliberately, with no mention of security, so it measures the unprompted setting. And one evaluated model, Qwen3.8-Max, also drafted the tests and specifications. The authors report it lands mid-field and see no sign this helped it, but it's an overlap worth keeping in mind.

Sam: So who should sit down with the full paper, and where should they start?

Alex: Anyone building, training, or evaluating coding agents, and security researchers. Start with the construction section, especially the behavioral attack tests and the comment-only decoy. Then read the analysis by weakness type and the security-reminder rerun. The case studies at the end are short and very concrete.

Sam: And for everyone else, the one line to carry around?

Alex: Code that passes its tests can still be wide open. Agents secure what good code naturally includes, not the policies nobody wrote down.

Sam: Worth remembering the next time a test suite turns green.