Natural generation allows Large Language Models (LLMs) to produce free-form responses with rich reasoning, yet the lack of structure makes outputs difficult to verify. Conversely, constrained decoding ensures standardized formats but can inadvertently restrict reasoning capabilities by imposing constraints too early in the generation process. We propose a hybrid approach, namely In-Writing, that combines free-form reasoning and structured generation in a single call. The model first performs unconstrained reasoning and only applies structured decoding after a trigger token is generated, explicitly decoupling reasoning from formatting. We establish that our trigger-token strategies are able to virtually eradicate premature triggering, a failure mode in which constrained decoding interrupts on-going reasoning. Evaluations across diverse datasets covering classification and reasoning tasks demonstrate that our approach outperforms the state-of-the-art by achieving accuracy gains of up to 27% over natural generation. Our code are available at: https://github.com/Nokia-Bell-Labs/InWriting.
Alex: Welcome to another episode of ResearchPod. Today, we're looking at a new way to help AI language models communicate more effectively with the software around them.
Sam: We're discussing a paper about a framework called "In-Writing." The puzzle it addresses is something that comes up constantly in real-world AI applications: how do you let a model think freely, while also making sure its final answer comes out in a strict, computer-readable format?
Alex: Why does that tension even exist? Can't the model just do both at once?
Sam: That's exactly the question. Here's the thing—these models generate language one word at a time, and the way they "think" is by writing out their reasoning as they go. If you force them to follow rigid formatting rules from the very first word, you're essentially interrupting that thinking process before it's had a chance to develop. But if you let them write freely, the output often looks nothing like what a database or software system needs to read.
Alex: It's like asking someone to brainstorm a solution to a complex problem, but forcing them to fill out an official form at the same time. The form gets in the way of the thinking.
Sam: That's a good way to put it. And the standard fix people use right now actually makes this worse in a subtle way. Most developers use something called "constrained decoding." Imagine a strict editor standing over the model's shoulder, checking every single word it writes against a rulebook. If the model tries to write something that doesn't fit the required structure, the editor blocks it immediately.
Alex: That sounds reasonable on the surface. Why is it a problem?
Sam: Because reasoning requires room to explore. When you block certain words too early, you're not just enforcing a format—you're cutting off entire lines of thought the model might need to reach the right answer. The researchers call this "premature triggering." The model gets interrupted mid-thought, loses its train of reasoning, and ends up with a worse answer even if the format looks correct.
Alex: Oh—so it's like a student who's working through a maths problem on scratch paper, but a teacher keeps grabbing the pen and writing the final answer before the student has actually finished working it out. The answer box looks tidy, but the work behind it was never completed.
Sam: That's exactly it. And that's the core insight the In-Writing framework is built around: the problem isn't the formatting rules themselves, it's *when* those rules kick in.
Alex: So how does In-Writing fix the timing?
Sam: The model is allowed to reason in plain, natural language for as long as it needs—no restrictions, no interruptions. It works through the problem step by step, which the researchers call "Chain-of-Thought." Think of it like showing your work in a maths class. Only once that reasoning is complete does the model emit what they call a "trigger token"—a specific signal, like a student moving from their scratch paper to the official answer box. The moment that signal appears, a different system takes over to enforce the strict format.
Alex: And that second system—how does it actually enforce the format?
Sam: It uses a technique called "Logit Masking." Every time the model is about to pick its next word, it's actually calculating a probability for every possible word in its vocabulary. Logit Masking works by taking any word that would violate the required format and setting its probability to zero—effectively erasing it as an option. The model can only choose from words that fit.
Alex: So it's not blocking the model from thinking—it's just quietly removing the wrong answers from the menu, but only at the very end.
Sam: Precisely. And because this only happens after the reasoning is already done, the model's thinking process is completely untouched. The paper suggests this is why the accuracy gains are meaningful—up to 27% better compared to systems that apply constraints from the very beginning.
Alex: That's a notable improvement. And it makes intuitive sense—you're not fighting the model's natural process, you're just redirecting it at the right moment.
Sam: That's the core insight. It's a shift from treating formatting as a constant restriction to treating it as a final, clearly-signalled step. The researchers tested this across 18 different models from five distinct model families, ranging from smaller to larger systems, and the results held up consistently.
Alex: Are there any limitations the paper flags?
Sam: There are. The paper notes that the system is sensitive to how you phrase the question you give it. The choice of trigger token and the structure of the prompt both matter—it's not quite "plug-and-play" yet. The researchers identify better prompt design as an area for future work.
Alex: So the underlying mechanism is solid, but there's still work to do on the user-facing side.
Sam: That's a fair summary. The separation of reasoning from formatting is a meaningful step toward making these tools more reliable in practice. The question of how to make that separation easy to configure is still open.
Alex: It's a useful reminder that even elegant solutions come with their own set of follow-on problems. Thanks for listening to ResearchPod.