Yitong Zhang, Shiteng Lu, Jia Li
5 min
Abstract
Large Language Models (LLMs) are increasingly used for code generation, raising concerns that they may be misused to produce malicious code. Meanwhile, Grammar-Constrained Decoding (GCD) has been widely adopted to improve the reliability of LLM-generated code by enforcing syntactic validity. In this paper, we reveal a counterintuitive risk: this reliability-oriented technique can itself become an attack surface. We uncover a new jailbreak attack, termed CodeSpear, that exploits GCD to induce LLMs into generating malicious code. Our experiments show that simply applying a benign code grammar constraint can effectively jailbreak LLMs. To address this vulnerability, we propose CodeShield, a safety alignment approach that robustly preserves safe behavior even under attacker-controlled grammar constraints. CodeShield aligns the model in the code modality by teaching it to generate honeypot code under GCD. Such code is semantically harmless, so it does not implement the malicious request, and structurally diverse, so it is difficult to suppress through grammar tightening. At the same time, CodeShield still preserves natural-language refusals when natural language is available. Experiments on 10 popular LLMs across 4 benchmarks show that CodeSpear outperforms representative jailbreak baselines and increases the attack success rate by more than 30 percentage points on average. CodeShield also restores safety under CodeSpear while preserving benign utility. Our findings reveal a fundamental risk of GCD and call for greater attention to its potential security implications.
Alex: So the AI can't say "no" in words, but it can say "no" in action—by writing code that goes nowhere.
Sam: Exactly. And the researchers made a specific design choice here that matters. The honeypot code is intentionally structurally diverse—meaning it varies a lot in how it looks. That's important because if the decoy code always followed the same pattern, an attacker could just add a rule to the grammar that filters it out. By making the honeypot unpredictable in structure, the defense becomes much harder to circumvent.
Alex: How do you actually teach the AI to do that? How do you train it to reach for the honeypot instead of the harmful code?
Sam: The researchers use a training technique called Direct Preference Optimization—DPO for short. Think of it like teaching someone good habits by showing them side-by-side choices. You show the AI two possible responses to a harmful request: one is the dangerous code, one is the honeypot. You repeatedly signal that the honeypot is the preferred choice. Over time, the AI internalizes that preference. It learns to reach for the safe decoy even when its usual English refusal tools have been taken away—and crucially, without losing its general ability to write useful code for legitimate requests.
Alex: So the broader lesson here is that AI safety can't just live in one place. If the safety training only works in English, then any tool that takes English off the table becomes a potential vulnerability.
Sam: That's the paper's central argument. As AI systems are increasingly deployed for specialized tasks—writing code, generating structured data, interacting with other software—the safety measures need to follow them into those specific contexts. A refusal mechanism that only works in natural language is a refusal mechanism with a known blind spot.
Alex: It's a useful reminder that building reliable AI isn't just about what the model can do—it's about making sure the safeguards hold up across every context it's used in. Thanks for listening to ResearchPod.