ResearchPod Summary
This paper introduces a controlled orchestration subsystem designed to automate incident response in educational information systems. As security alert volumes grow, manual review becomes unsustainable, yet automated responses risk disrupting legitimate academic services. The authors propose an architecture that integrates a Rule Engine, a local Large Language Model (LLM), and a durable event queue to manage responses safely.
The system operates on a clear separation of concerns. A deterministic Rule Engine handles severity classification and playbook selection, ensuring that technical decisions remain predictable and auditable. The LLM is relegated to an advisory role, providing contextual analysis that is processed through a rigorous pipeline of validators, guardrails, and output sanitizers. This ensures that even if the model provides incomplete or hallucinated advice, the system defaults to safe, predefined operational policies. The architecture also includes a durable SQLite-based queue to prevent duplicate tasks and ensure recovery from interruptions without re-executing firewall rules.
Experiments conducted in a laboratory environment confirmed that the system maintains functional correctness and safety. The Rule Engine consistently matched predefined routing matrices, and the durable queue successfully processed 100 events without errors or duplicate actions. While the LLM provided useful advisory context, the study found that local inference was the primary processing bottleneck, adding significant latency compared to the deterministic baseline. The authors conclude that for well-defined alert categories, deterministic processing is superior, while LLMs should be reserved for complex, novel scenarios requiring deeper contextual synthesis.
This research provides a blueprint for integrating generative AI into security operations without sacrificing control. By treating LLMs as untrusted advisory components rather than autonomous agents, organizations can leverage AI for analysis while maintaining the strict safety and auditability requirements necessary for critical network infrastructure.
[[RP_SECTION:deterministic-rule-engine-architecture|Deterministic Rule Engine Architecture]]
Alex: Deterministic rule engines beat mandatory LLM integration for handling security alerts. That's the headline from a study by Hoang-Lam Huynh and colleagues on post-alert orchestration — and the reason is almost the opposite of what you'd expect from an AI paper. The LLM does better work when it's stripped of authority entirely.
Sam: So the model doesn't get to decide anything. How do they stop it from hallucinating a recommendation that could take down a server, or worse, tell an admin to block the wrong traffic?
Alex: They isolate authority structurally. A deterministic Rule Engine handles severity classification and playbook selection — that's the part that actually touches the firewall. The LLM's only job is generating explanatory text for a human, and even that has to pass a strict validation layer before it reaches an administrator. If the output fails validation, the system doesn't retry or improvise — it falls back to a static, pre-verified record. [[RP_SECTION:reliability-and-functional-correctness|Reliability and Functional Correctness]]
Sam: That's a clean separation of concerns. What did the reliability numbers actually look like once they tested the boundaries?
Alex: In boundary testing, the rule engine hit perfect functional correctness — every alert got routed to the right playbook, no exceptions. They backed that with a durable SQLite queue, so a restart mid-processing can't cause an event to be silently dropped or processed twice. That's the load-bearing result here: the deterministic core is doing all the heavy lifting on correctness, and the LLM isn't part of that guarantee at all.
Sam: So the LLM is essentially an intern drafting a report that a senior engineer has to sign off on before it goes anywhere. What did that validation layer cost in terms of speed? [[RP_SECTION:latency-and-auditability-tradeoffs|Latency and Auditability Tradeoffs]]
Alex: Mean processing time came in around thirty-three seconds. Not instantaneous, but the point isn't speed — it's auditability. The model never touches the firewall configuration; it only supplies context for a human in the loop. That's a deliberate trade: you're paying latency to keep every decision traceable back to a deterministic rule rather than a model inference you can't fully explain after the fact. [[RP_SECTION:system-performance-under-load|System Performance Under Load]]
AI-generated third-party summary by ResearchPod. Not official content or an endorsement by the paper authors or affiliated organizations.
Sam: What happens under load, though? If eight alerts fire at once and the LLM is generating explanations for all of them, does the whole pipeline stall waiting on inference?
Alex: They tested exactly that — eight simultaneous alerts. The architecture caps the LLM at one active request at a time. Everything else queues, but critically, the deterministic rule processing doesn't wait on the model — it keeps working through the backlog regardless of how many LLM calls are pending. That's the supporting evidence for the main claim: the system's core function survives contention even when the generative component becomes a bottleneck. [[RP_SECTION:limitations-of-static-design|Limitations of Static Design]]
Sam: Is there a scenario where the static design actually falls short — where the safety comes at a real cost?
Alex: Yes, and the authors are upfront about it. Their retrieval layer — a Static RAG setup — depends on manually maintained category mappings for known alert types. If an attack pattern shows up that isn't already in that versioned knowledge base, the system has no mechanism to adapt on its own. It's not going to misfire dangerously, but it also won't recognize something genuinely novel. That's the real ceiling on this architecture, not the LLM's reliability.
Sam: So you're trading the flexibility of a generative system for the predictability of a deterministic one. For a university network where stability matters more than catching zero-days on the fly, that seems like a defensible trade.
Alex: That's the paper's framing too. And it's worth being precise about what the evidence supports here. This was tested in a lab environment with synthetic alerts — not live production traffic, and not at the scale of a large distributed network. The authors don't claim to have solved real-time throughput or scaling; what they've shown is that when you treat an LLM as an untrusted, advisory-only component and force every output through validation, you get a system whose security posture stays predictable even when the model fails or hallucinates.
Sam: That's the part worth sitting with — it's a rebuttal to the assumption that reasoning capability should translate into decision-making authority. Just because a model can generate a plausible next step doesn't mean it should be the one taking it. [[RP_SECTION:future-research-directions|Future Research Directions]]
Alex: Exactly the trap this design avoids. The next step the authors point to is moving from those static category mappings toward something like a dynamic knowledge graph — a way to let the system recognize unfamiliar alert patterns without giving up the deterministic guarantees that make it auditable in the first place. Whether that's achievable without reintroducing the very unpredictability they engineered away is the open question.
Sam: A reasonable place to leave it — deterministic first, generative only where it can't do damage.
Alex: That's the shape of it. Thanks for listening to ResearchPod.