AutoSpec: Safety Rule Evolution for LLM Agents via Inductive Logic Programming

LLM agents are increasingly used to automate complex tasks, but their autonomy introduces significant safety risks. While expert-designed symbolic rules are preferred for their interpretability and auditability, they are often brittle and struggle to adapt to evolving agent behaviors, leading to high false-positive or false-negative rates. This paper asks: can we automatically evolve these static, hand-crafted safety rules using user-provided feedback to maintain high precision and recall without sacrificing interpretability?

AutoSpec frames the maintenance of safety rules as a Counterexample-Guided Inductive Synthesis (CEGIS) problem. It treats user-annotated safe and unsafe execution traces as counterexamples that reveal where existing rules fail.

To avoid the exponential search space inherent in rule synthesis, AutoSpec employs Inductive Logic Programming (ILP). In each iteration, the system:

1. Mines false positives and false negatives from current rule performance.
2. Uses an ILP solver to identify which predicates best discriminate between these safe and unsafe traces.
3. Generates candidate rule edits based on these prioritized predicates.
4. Verifies the candidates and selects the best revision to improve the overall F1 score.

This loop continues until the rules converge or a target performance threshold is met, effectively automating the maintenance process that typically requires manual expert intervention.

The authors evaluated AutoSpec on 291 execution traces across code execution and embodied agent domains. The system achieved F1 scores of 0.98 and 0.93, respectively, and reduced false positives by up to 94%. By using ILP to guide the search, AutoSpec converged within 4–5 iterations and outperformed heuristic-based CEGIS approaches by up to 4.8x in F1 score. The resulting rules remained human-readable and generalized well to unseen tasks, demonstrating that symbolic safety layers can be effectively maintained through automated, data-driven evolution.