Ngoc Trinh Hung Nguyen, Alonso Silva, Laith Zumot, Liubov Tupikina, Armen Aghasaryan, Mehwish Alam
5 min
Abstract
Natural generation allows Large Language Models (LLMs) to produce free-form responses with rich reasoning, yet the lack of structure makes outputs difficult to verify. Conversely, constrained decoding ensures standardized formats but can inadvertently restrict reasoning capabilities by imposing constraints too early in the generation process. We propose a hybrid approach, namely In-Writing, that combines free-form reasoning and structured generation in a single call. The model first performs unconstrained reasoning and only applies structured decoding after a trigger token is generated, explicitly decoupling reasoning from formatting. We establish that our trigger-token strategies are able to virtually eradicate premature triggering, a failure mode in which constrained decoding interrupts on-going reasoning. Evaluations across diverse datasets covering classification and reasoning tasks demonstrate that our approach outperforms the state-of-the-art by achieving accuracy gains of up to 27% over natural generation. Our code are available at: https://github.com/Nokia-Bell-Labs/InWriting.
Alex: And that second system—how does it actually enforce the format?
Sam: It uses a technique called "Logit Masking." Every time the model is about to pick its next word, it's actually calculating a probability for every possible word in its vocabulary. Logit Masking works by taking any word that would violate the required format and setting its probability to zero—effectively erasing it as an option. The model can only choose from words that fit.
Alex: So it's not blocking the model from thinking—it's just quietly removing the wrong answers from the menu, but only at the very end.
Sam: Precisely. And because this only happens after the reasoning is already done, the model's thinking process is completely untouched. The paper suggests this is why the accuracy gains are meaningful—up to 27% better compared to systems that apply constraints from the very beginning.
Alex: That's a notable improvement. And it makes intuitive sense—you're not fighting the model's natural process, you're just redirecting it at the right moment.
Sam: That's the core insight. It's a shift from treating formatting as a constant restriction to treating it as a final, clearly-signalled step. The researchers tested this across 18 different models from five distinct model families, ranging from smaller to larger systems, and the results held up consistently.
Alex: Are there any limitations the paper flags?
Sam: There are. The paper notes that the system is sensitive to how you phrase the question you give it. The choice of trigger token and the structure of the prompt both matter—it's not quite "plug-and-play" yet. The researchers identify better prompt design as an area for future work.
Alex: So the underlying mechanism is solid, but there's still work to do on the user-facing side.
Sam: That's a fair summary. The separation of reasoning from formatting is a meaningful step toward making these tools more reliable in practice. The question of how to make that separation easy to configure is still open.
Alex: It's a useful reminder that even elegant solutions come with their own set of follow-on problems. Thanks for listening to ResearchPod.