ResearchPod Summary
As domain-specific languages (DSLs) become essential for managing operating system security and infrastructure, the manual creation of rules has become a major source of configuration errors. This paper investigates whether large language models (LLMs) can automate the generation of DSL code from natural language (Text2DSL) and identifies the most effective prompting strategies to ensure the generated code is both syntactically and semantically correct.
The authors formalize the Text2DSL task, distinguishing it from general-purpose code generation by its reliance on a fixed, compact grammar and a closed vocabulary of identifiers. To evaluate this, they introduce PolkitBench, a new dataset containing 4,204 verified natural-language-to-Polkit-rule pairs. The researchers then conducted controlled experiments on two Mixture-of-Experts (MoE) models, comparing a baseline prompt against a context-enhanced prompt that explicitly provides the target DSL's BNF grammar, API specifications, and permitted identifier vocabulary.
The study demonstrates that structured context is a critical enabler for high-quality DSL code generation. Across both tested models, providing formal context increased syntactic validity to nearly 100% (98.6–99.4%) and significantly improved structural validity and CodeBLEU scores (by 60% to 95%). These results suggest that for DSLs, the model's performance is constrained more by the lack of explicit, formal constraints in the prompt than by the model's inherent size or training data. By injecting the target language's formal specification, the models were able to avoid common pitfalls like API hallucinations and invalid identifier usage, achieving high performance without the need for expensive fine-tuning.
This research provides a scalable framework for automating the configuration of security-critical systems. By treating DSL generation as a distinct class of problem, the authors show that developers can achieve reliable, verifiable code generation for specialized tools—such as Polkit, SELinux, or Kubernetes admission controllers—simply by providing the model with the language's formal grammar. This approach lowers the barrier to entry for managing complex infrastructure and reduces the risk of security misconfigurations caused by human error.
AI-generated third-party summary by ResearchPod. Not official content or an endorsement by the paper authors or affiliated organizations.