Large Language Models (LLMs) are increasingly used for code generation, raising concerns that they may be misused to produce malicious code. Meanwhile, Grammar-Constrained Decoding (GCD) has been widely adopted to improve the reliability of LLM-generated code by enforcing syntactic validity. In this paper, we reveal a counterintuitive risk: this reliability-oriented technique can itself become an attack surface. We uncover a new jailbreak attack, termed CodeSpear, that exploits GCD to induce LLMs into generating malicious code. Our experiments show that simply applying a benign code grammar constraint can effectively jailbreak LLMs. To address this vulnerability, we propose CodeShield, a safety alignment approach that robustly preserves safe behavior even under attacker-controlled grammar constraints. CodeShield aligns the model in the code modality by teaching it to generate honeypot code under GCD. Such code is semantically harmless, so it does not implement the malicious request, and structurally diverse, so it is difficult to suppress through grammar tightening. At the same time, CodeShield still preserves natural-language refusals when natural language is available. Experiments on 10 popular LLMs across 4 benchmarks show that CodeSpear outperforms representative jailbreak baselines and increases the attack success rate by more than 30 percentage points on average. CodeShield also restores safety under CodeSpear while preserving benign utility. Our findings reveal a fundamental risk of GCD and call for greater attention to its potential security implications.
Alex: Welcome to another episode of ResearchPod. Today, we're looking at a paper that explores a surprising vulnerability in how we use AI to write computer code.
Sam: We're discussing a study on something called Grammar-Constrained Decoding, or GCD. To understand it, think about how AI coding tools work. When you ask an AI to write you a Python script, it doesn't just generate any text—it's guided by a set of rules that ensure the output is valid, runnable code. That rule-enforcing system is GCD. The problem the researchers found is that this same tool can inadvertently bypass the AI's safety filters.
Alex: So a tool meant to make AI-written code more reliable is accidentally making it less safe?
Sam: That's exactly the tension the paper explores. To understand why, you need to know how AI safety currently works. Right now, when an AI is asked to do something harmful—say, write malware—it's trained to respond in plain English with a refusal, something like "I can't help with that." The safety system lives entirely in natural language.
Alex: Okay. So if the safety training is all in English, what happens when you force the AI to stop speaking English entirely?
Sam: That's the crux of it. When you apply GCD, you're essentially putting the AI in a room where it's only allowed to speak in the grammar of a programming language. It cannot produce a sentence like "I cannot assist with that"—that's not valid Python. So when someone asks it to do something harmful, it can no longer reach for its usual refusal. It's left with only one option: write code.
Alex: So the safety guardrail isn't broken—it's just been silenced. The AI still "wants" to refuse, but the rules of the room won't let it say the words.
Sam: That's a good way to put it. The researchers named this attack "CodeSpear." What makes it notable is how simple it is. You don't need to do any sophisticated hacking. You just apply a standard, ordinary Python grammar to the AI's output, and the refusal mechanism becomes unreachable. The AI, finding no valid way to decline, defaults to doing what was asked.
Alex: That's a meaningful gap. How do you even begin to fix something like that?
Sam: The authors propose a defense called "CodeShield," and the core idea is clever. Rather than trying to teach the AI to refuse in code—which is hard, because "refusal" doesn't really have a clean equivalent in Python—they train it to generate what they call "honeypot code."
Alex: Honeypot code? That sounds like a trap.
Sam: It is, in a sense. Honeypot code is code that looks structurally valid—it follows all the grammar rules, so GCD accepts it—but it doesn't actually do anything harmful. It's a decoy. The AI produces something that satisfies the constraint of "you must output valid code" while completely failing to carry out the attacker's request.
Alex: So the AI can't say "no" in words, but it can say "no" in action—by writing code that goes nowhere.
Sam: Exactly. And the researchers made a specific design choice here that matters. The honeypot code is intentionally structurally diverse—meaning it varies a lot in how it looks. That's important because if the decoy code always followed the same pattern, an attacker could just add a rule to the grammar that filters it out. By making the honeypot unpredictable in structure, the defense becomes much harder to circumvent.
Alex: How do you actually teach the AI to do that? How do you train it to reach for the honeypot instead of the harmful code?
Sam: The researchers use a training technique called Direct Preference Optimization—DPO for short. Think of it like teaching someone good habits by showing them side-by-side choices. You show the AI two possible responses to a harmful request: one is the dangerous code, one is the honeypot. You repeatedly signal that the honeypot is the preferred choice. Over time, the AI internalizes that preference. It learns to reach for the safe decoy even when its usual English refusal tools have been taken away—and crucially, without losing its general ability to write useful code for legitimate requests.
Alex: So the broader lesson here is that AI safety can't just live in one place. If the safety training only works in English, then any tool that takes English off the table becomes a potential vulnerability.
Sam: That's the paper's central argument. As AI systems are increasingly deployed for specialized tasks—writing code, generating structured data, interacting with other software—the safety measures need to follow them into those specific contexts. A refusal mechanism that only works in natural language is a refusal mechanism with a known blind spot.
Alex: It's a useful reminder that building reliable AI isn't just about what the model can do—it's about making sure the safeguards hold up across every context it's used in. Thanks for listening to ResearchPod.