AI coding agents now write much of the code that ships, yet whether that code is secure remains unclear. We introduce the Last Secure Code Benchmark, a comprehensive evaluation of secure code generation by coding agents. Built from 150 real vulnerabilities, its 450 tasks span seven widely used languages (Java, C, C++, JavaScript, Python, PHP, and Go), 24 CWEs including 21 of the 2025 CWE Top 25 and 8 of the Top 10 KEV weaknesses, more than any benchmark we compare with, and three vibe-coding scenarios: filling in a function, writing a file, and building a repository's source tree from scratch. Every task asks only for functionality, through a specification that never mentions security, and hidden functional and security tests grade the result. Across six frontier agents, 73.6% to 91.3% of tasks pass functionally, yet even the strongest, Claude Fable 5, produces code that is both functional and secure for only 25.1% of tasks, and the others for 16.4% to 24.7%. The best model only passes 12% of the vulnerability scenarios across all vibe coding setting, posing strong concern on the security of the code written by vibe coding.
Alex: Ask an AI agent for working code, and you'll usually get it. Ask whether that code is also secure, and the answer is usually no. Across six frontier coding agents, the best one wrote code that was both working and secure on only about a quarter of tasks.
Sam: Before we dig into that quarter, tell me where it comes from. Whose work is this, and what kind of study is it?
Alex: It's a benchmark called the Last Secure Code Benchmark, from researchers at the National University of Singapore and teams at Alibaba, including the Qwen team. It's an arXiv preprint from October 2026. They built 450 tasks from 150 real vulnerabilities, across seven languages.
Sam: A lot of people now write software by just describing what they want. Why should someone outside security care about this result?
Alex: Because that workflow, often called vibe coding, has a blind spot. You describe what the software should do, the agent writes it, and you check that it works. You rarely read the code. So security decisions, like how to handle untrusted input, get made silently. We'll cover how they tested this, what they found, and how far to trust it.
Sam: There are already security benchmarks for generated code. What was missing?
Alex: The authors argue each earlier one falls short somewhere. Some only ask for a short snippet. Some cover one or two languages. Some reach only a slice of the important weakness types. This one covers 24 categories from the CWE list, the standard catalogue of software weakness types. And it tests three scopes of work: filling in one function, writing a whole file, or rebuilding a chunk of a repository from scratch.
Sam: Security tests are easy to get wrong, though. How do they know a test is really checking security and not just looking for the right keyword?
Alex: That's the heart of the method. They found the upstream tests often inspected source text. A solution could pass by adding the right comment. So experts rewrote every test to run a proof-of-concept attack, a real exploit, against the running program. A test was kept only if it caught the original vulnerable code and passed the fixed code.
Sam: And the comment trick specifically, did they guard against that?
Alex: They did. For each task, they built a decoy: the vulnerable code with every line of the fix pasted in as comments. The program behaves exactly as before, but its text contains the whole fix. If a security test passed that decoy, it was matching words, and it was thrown out.
Sam: Make the setup concrete for me. What does an agent actually get?
Alex: Take one of their Java tasks. A cloud platform has a service that mounts storage by running the system mount command, built from stored file paths. The agent gets the project with that service file removed, plus a long specification of what the service must do. The specification never mentions security. Hidden tests then check that mounting works, and try storing a path with a shell command tacked onto the end.
Sam: So what separates a pass from a fail there?
Alex: One choice the specification leaves open: how to launch the process. Four of the six models passed the arguments as a list, so no shell ever interprets the paths. Claude Fable 5 even noted unprompted that this stops injection. The other two built a command string for the shell. Same specification, same functional result, and the injection stayed open.
Sam: So functionally they're indistinguishable. Only the attack test sees the difference.
Alex: That's exactly the gap the benchmark is built to expose.
Sam: Then give me the headline numbers, and what they're compared against.
Alex: Five of the six models passed the functional tests on about nine in ten tasks. But the best, Claude Fable 5, was both functional and secure on about 25 percent. The others ranged down to about 16 percent. The difference is almost all one outcome: code that works and keeps the vulnerability.
Sam: Does a model at least handle a given vulnerability consistently? If it secures the function version, does it secure the file and repository versions too?
Alex: Rarely. The best model secured all three versions of a scenario on only 12 percent of scenarios. So securing it once is no evidence the model understood it.
Sam: I'd guess bigger tasks are harder to secure. More code, more places to slip.
Alex: The document shows something different. Functional completion falls as scope grows. But functional-and-secure is flat or rising from function to repository for every model. The authors offer a possible reason: filling a hole inside existing code invites the agent to reproduce the original vulnerable design. Building a module from scratch, it picks the checks itself.
Sam: Which kinds of weaknesses do the agents get right, and which do they miss?
Alex: Memory safety is secured most often, roughly a third of attempts pooled across models. Those defenses are local, like a bounds check, and careful code includes them anyway. At the bottom are weaknesses whose defense is a policy nobody stated. Two are almost never secured. One is information exposure. The other is server-side request forgery, tricking a server into fetching things it shouldn't.
Sam: Is there a case that shows the policy problem clearly?
Alex: A tool called syft prints its configuration, which includes an attestation password. The agent had to rewrite a small config struct. Every model copied the tagging style of the neighboring fields, so the password got printed with everything else. No trajectory mentioned that the value was a password. The authors' point is that the check is easy to write but hard to know you need.
Sam: Surely just telling the agent to be secure fixes a lot of this.
Alex: Some of it. They added one sentence asking for secure code and reran four models. For three of them, functional-and-secure rose by roughly 15 to 21 points, with functional pass rates barely moving. Most tasks still stayed insecure, though. Information exposure and request forgery stayed near the bottom even with the reminder.
Sam: What about effort? Would more turns, more checking, buy more security?
Alex: Not in this data. The model using the fewest turns secured the most tasks. Others spent about twice as many turns and secured fewer. In one case, an agent wrote its own smoke test and grew it to thirty-three checks. It caught a corrupt-archive edge case. But every check used well-formed input. Its YAML parser, for reading uploaded config files, would build any class an attacker named. Those were the same hidden attacks two other models blocked with a safe parser.
Sam: Different models fail differently, then. Does pooling them help?
Alex: Pooled, at least one model secured about 44 percent of tasks, against 25 for the best alone. But the authors stress that union is picked in hindsight, so it's not a strategy you could run in advance. And more than half the tasks, 56 percent, were secured by no model at all.
Sam: What should make me cautious about these numbers?
Alex: A few things. Each task is attempted once, deliberately, with no mention of security, so it measures the unprompted setting. And one evaluated model, Qwen3.8-Max, also drafted the tests and specifications. The authors report it lands mid-field and see no sign this helped it, but it's an overlap worth keeping in mind.
Sam: So who should sit down with the full paper, and where should they start?
Alex: Anyone building, training, or evaluating coding agents, and security researchers. Start with the construction section, especially the behavioral attack tests and the comment-only decoy. Then read the analysis by weakness type and the security-reminder rerun. The case studies at the end are short and very concrete.
Sam: And for everyone else, the one line to carry around?
Alex: Code that passes its tests can still be wide open. Agents secure what good code naturally includes, not the policies nobody wrote down.
Sam: Worth remembering the next time a test suite turns green.