ResearchPod Summary
This paper addresses the challenge of generating domain-specific language (DSL) code from natural language (Text2DSL). The authors extend their previous work by introducing a context-aware distillation method to create a high-quality, verified corpus (PolkitBench-v2) and performing a factorial ablation study to determine which components of structured context—BNF grammar, API specifications, or identifier vocabularies—are most critical for model performance.
To build the corpus, the authors used DeepSeek-V4-Flash as a teacher model, constrained by a formal context model. Every generated rule underwent a two-tier validation process: static AST validation using esprima and runtime verification by executing the rules within a production polkitd daemon. This resulted in a verified dataset of 10,073 rule pairs.
The authors evaluated the GigaChat-10B-A1.8B model on this new, harder corpus. They found that without structured context, the model's performance collapsed significantly (Combined Score dropping from 0.482 to 0.252). In contrast, models provided with full structured context maintained high performance, confirming that context is not merely a cosmetic prompt addition but a necessary component for reliable code generation.
The factorial ablation (testing all combinations of BNF, API, and Vocabulary) revealed that the identifier vocabulary is the most significant contributor to semantic quality. Meanwhile, the BNF grammar and API specifications are the primary drivers for ensuring structural and syntactic validity. The full-context condition (using all three) consistently outperformed all partial configurations.
For practitioners building automated policy or configuration tools, this research provides a clear roadmap for prompt engineering and data generation. It demonstrates that simply scaling model size is insufficient for domain-specific tasks; instead, explicitly binding the model to the formal constraints of the target language (the context model) is essential for achieving production-grade reliability. The use of a two-tier validation pipeline also sets a standard for ensuring that generated code is not just syntactically correct, but functionally executable in real-world environments.
AI-generated third-party summary by ResearchPod. Not official content or an endorsement by the paper authors or affiliated organizations.