Cheng Meng, Wenxin Le, Xinyi Li, Qiuyun Wang, Fangli Ren, Zhengwei Jiang, Baoxu Liu
10 min
Cybersecurity relies on detection rules—formal specs that translate threat behaviors into executable logic for tools like Snort (network IDS), Splunk (SIEM), or Sigma (universal format). Think of rules as the 'if-then' logic that flags malware process execution or suspicious network connections. But current automated generation is fragmented: one pipeline for CTI reports to Sigma rules, another for network traffic to Suricata, etc. Each combo needs custom engineering, limiting scalability across diverse threats, inputs, and platforms.
The paper formalizes rule generation as a single function f: C × L → R:
Optimal rule r* minimizes semantic distance: symmetric difference between what it covers Cov(r) and the intended threat behaviors I(c) ∩ E(l) (intersection of context intent and language expressiveness). This shifts evaluation from token similarity to semantic equivalence, enabling cross-language comparison.
UniRule is a multi-agent RAG system that unifies generation:
This decouples retrieval from syntax, enabling reuse across languages/contexts in one system.
Tested on 12 scenarios (3 languages × 4 contexts), 12k pairwise prefs:
Key insight: Semantic distance validates projections as the right abstraction for 'optimal rules.'
UniRule pioneers studying rule gen as a unified task, not scenario-specific hacks. It scales to the 'threat behavior universe' (atomic observables like 'process spawn'), paving for general-purpose cybersecurity automation. For students: bridges NLP/LLMs with security ops, showing RAG + formalization tackles real fragmentation.
Existing methods for detection rule generation are tightly coupled to specific input-output combinations, requiring dedicated pipelines for each. We formalize this problem as a unified mapping f:C*L->R and characterize optimal rules through semantic distance. We propose UniRule, an agentic RAG framework built on dual semantic projection spaces: detection intent and detection logic. This design enables retrieval and generation across arbitrary contexts and target languages within a single system. Experiments across 12 scenarios (3 languages, 4 context types, 12,000 pairwise comparisons) show that UniRule significantly outperforms pure LLM generation with a Bradley-Terry coefficient of 0.52, validating semantic projection as an effective abstraction for unified rule generation. Together, the formalization, method, and evaluation provide an initial framework for studying detection rule generation as a unified task.
Alex: So it's like converting a bunch of foreign cookbooks into English summaries of purpose and steps, then filing them for quick lookup. Okay—and then online, when a new threat description comes in, how does the system pull from that?
Sam: In the online phase, an LLM acts as an agent to generate the new rule. It reads the input threat context and target language, then dynamically searches the indexes—deciding on its own which space to query, intent or logic, how many similar items to fetch, and even whether to filter by language. It might do this over several rounds, refining as it goes, before combining the retrieved pieces with the input to output a rule in the right format. This agentic retrieval-augmented generation lets it adapt to whatever the context needs, pulling relevant patterns across languages without rigid paths.
Alex: Wait—dynamically deciding which space and how many? That sounds flexible, but how do they know if the generated rules are actually good?
Sam: They define rule quality by comparing what a rule actually catches to what it should catch, given the threat description and the limits of the target language. Imagine all possible bad actions as a big set of dots; the ideal rule covers exactly the right ones without extras or misses—that mismatch is the symmetric difference. Since exact math is hard without test data, they use pairwise checks: experts or models pick which of two rules better matches the goal, then aggregate those into a score. The paper shows UniRule wins these head-to-heads against baselines in most cases.
Alex: Huh—so quality isn't about matching one example perfectly, but getting closer to the ideal coverage no matter the surface details. Right, that pairwise approach feels solid for judging rules. But how did they actually set up the tests to check if UniRule holds up across real variety?
Sam: They tested on three common rule languages—Splunk for log analysis, Elastic for endpoint monitoring, and Snort for network threats—pulling rules from public collections and splitting them into training and test sets. Each test rule got paired with four types of input: the original description, a simulated threat report, a high-level goal like the attack's aim, or a detailed spotting method. This made twelve combinations to cover different ways threats get described. The goal was to see if one system could handle them all without custom tweaks.
Alex: So it's like testing a translator on short notes, full stories, goals, and instructions across three dialects. Makes sense for practicality. And the agent part—how does it decide what to pull during generation?
Sam: The agent, powered by the language model, doesn't follow a fixed script. Instead, it reads the input and target language, then chooses on its own: skip retrieval if the input's detailed enough, pick intent or logic space or both, craft the search query, set how many matches to fetch, and even loop back for more rounds if needed. For vague inputs like brief reports, it digs deeper across spaces; for precise ones, it generates straight away. This flexibility adapts to the job without hard rules.
Alex: Okay—that sounds smarter than rigid pipelines. But to measure quality without running every rule on real threats, what trick did they use? And how did UniRule stack up?
Sam: They skipped surface matches like word overlap, since good rules can look totally different but catch the same bad actions. Instead, for each test, the system made two rules, and a judging model picked the better one by how well it matched the threat goal without extras or misses—doing this thousands of times across pairs. Those choices get combined into a single score using a ranking method. Overall, UniRule came out ahead, winning nine of twelve scenarios—even against random or standard retrieval.
Alex: Huh. So it's like tournament rankings from head-to-heads, not total scores. That edge is notable, especially winning most scenarios. But why the variation across languages like Splunk versus Snort?
Sam: It boils down to what each language detects. Splunk and Elastic rules often track patterns like event counts or field matches in logs and endpoints, where pulling similar threat goals or spotting methods from other rules fills useful gaps. Snort focuses on exact network signatures, like specific byte sequences in traffic—those details don't transfer well semantically, so retrieval adds noise instead of help. The paper notes this as a limit when inputs already spell out everything or when low-level specifics can't be inferred.
Alex: So retrieval shines when there's room to borrow ideas, but backfires on hyper-specific matches. What about testing just one space at a time?
Sam: They checked that by running versions using only intent descriptions or only logic ones. Each alone beat the plain language model baseline, showing either space adds value. Combining them gave a small extra lift overall, because the two descriptions often overlap—they describe the same rule from slightly different angles, pulling similar examples anyway.
Alex: Huh, so redundancy between spaces explains the modest combo gain. Got an example of it working in practice?
Sam: Figure four shows a case: given a description of users hammering a system with huge token counts and requests—maybe to crash it or steal data—the agent queries both spaces. It pulls intent matches on resource exhaustion goals and logic ones on high-volume monitoring, then weaves them into an Elastic rule with time buckets, multi-thresholds, and severity scores. That output covers more ground than a human-written one, which skipped those extras.
Alex: That's a clear step up in completeness. But the paper mentions downsides—any caveats on when retrieval hurts?
Sam: Yes, it's a double-edged tool: great for gaps, like vague threat reports needing pattern ideas, but counterproductive for complete specs or non-transferable details. On Snort with logic inputs, it underperformed because the input already had the full blueprint, and added references just muddied things. The authors suggest future work could detect those cases to skip retrieval, and real traffic tests would confirm coverage in the wild. Human-written rules often score lower too—these expert rules are tuned for real-world use, short and simple to avoid glitches, assuming other tools handle the basics. But the tests judge how fully a rule matches the described threat goal, so fuller rules from the system edge them out there.
Alex: Right—so the scoring favors completeness over streamlined deployment. That gap between lab tests and operations makes sense as a caution. Overall, UniRule takes a meaningful step toward one system handling diverse threat descriptions into any rule format.
Sam: The evidence supports that—stronger preferences in most of the twelve tested mixes. It shows unified generation is workable, paving the way for quicker defenses without custom builds for every tool. With refinements like skipping retrieval when unneeded, and testing on live traffic, it could lead to rules ready for production at scale.
Alex: That's a solid foundation, balancing what's achieved with clear next hurdles. Thanks, Sam—this has clarified how such a system could streamline cybersecurity.
Sam: My pleasure, Alex. Thanks for listening to ResearchPod.