Design and Evaluation of a Controlled Post-Alert Incident Orchestration and Response Subsystem Using a Rule Engine and a Local Large Language Model | Hoang-Lam Huynh et al. | ResearchPod

Transcript: Design and Evaluation of a Controlled Post-Alert Incident Orchestration and Response Subsystem Using a Rule Engine and a Local Large Language Model

[[RP_SECTION:deterministic-rule-engine-architecture|Deterministic Rule Engine Architecture]]

Alex: Deterministic rule engines beat mandatory LLM integration for handling security alerts. That's the headline from a study by Hoang-Lam Huynh and colleagues on post-alert orchestration — and the reason is almost the opposite of what you'd expect from an AI paper. The LLM does better work when it's stripped of authority entirely.

Sam: So the model doesn't get to decide anything. How do they stop it from hallucinating a recommendation that could take down a server, or worse, tell an admin to block the wrong traffic?

Alex: They isolate authority structurally. A deterministic Rule Engine handles severity classification and playbook selection — that's the part that actually touches the firewall. The LLM's only job is generating explanatory text for a human, and even that has to pass a strict validation layer before it reaches an administrator. If the output fails validation, the system doesn't retry or improvise — it falls back to a static, pre-verified record. [[RP_SECTION:reliability-and-functional-correctness|Reliability and Functional Correctness]]

Sam: That's a clean separation of concerns. What did the reliability numbers actually look like once they tested the boundaries?

Alex: In boundary testing, the rule engine hit perfect functional correctness — every alert got routed to the right playbook, no exceptions. They backed that with a durable SQLite queue, so a restart mid-processing can't cause an event to be silently dropped or processed twice. That's the load-bearing result here: the deterministic core is doing all the heavy lifting on correctness, and the LLM isn't part of that guarantee at all.

Sam: So the LLM is essentially an intern drafting a report that a senior engineer has to sign off on before it goes anywhere. What did that validation layer cost in terms of speed? [[RP_SECTION:latency-and-auditability-tradeoffs|Latency and Auditability Tradeoffs]]

Alex: Mean processing time came in around thirty-three seconds. Not instantaneous, but the point isn't speed — it's auditability. The model never touches the firewall configuration; it only supplies context for a human in the loop. That's a deliberate trade: you're paying latency to keep every decision traceable back to a deterministic rule rather than a model inference you can't fully explain after the fact. [[RP_SECTION:system-performance-under-load|System Performance Under Load]]

Sam: What happens under load, though? If eight alerts fire at once and the LLM is generating explanations for all of them, does the whole pipeline stall waiting on inference?

Design and Evaluation of a Controlled Post-Alert Incident Orchestration and Response Subsystem Using a Rule Engine and a Local Large Language Model

Hoang-Lam Huynh, Quoc-Cuong Tang, Van-Tri Phan, Khuong Nguyen-An

Sep 23, 2026·4:22·arxiv·Original Paper

Security OrchestrationRule EngineStatic RAGLocal Large Language ModelPost-Alert Incident ResponseSecurity Orchestration, Automation, and Response (SOAR)

Listen

4 min

Briefing

Abstract

This paper presents a controlled post-alert incident orchestration and response subsystem for educational information systems. The architecture separates deterministic classification, contextual analysis, human approval, and technical execution. A Rule Engine determines severity and selects the playbook, while Static RAG and a local large language model provide advisory content under Validator, Guardrail, Output Sanitizer, and Safe Fallback controls. Experiments begin after simulated alerts are stored in Elasticsearch. The Rule Engine matched the predefined routing matrix in all 30 boundary cases. The Durable Queue completed 100 events without duplicate tasks, new failed tasks, or unintended firewall rules. An eight-alert contention experiment preserved the configured limit of one active model request, and 30 sequential measurements showed an overall mean post-alert processing time of approximately 33 seconds. The results demonstrate functional correctness, traceability, controlled recovery, and bounded model integration within the evaluated laboratory scope.

Overview

This paper introduces a controlled orchestration subsystem designed to automate incident response in educational information systems. As security alert volumes grow, manual review becomes unsustainable, yet automated responses risk disrupting legitimate academic services. The authors propose an architecture that integrates a Rule Engine, a local Large Language Model (LLM), and a durable event queue to manage responses safely.

Architecture and Approach

The system operates on a clear separation of concerns. A deterministic Rule Engine handles severity classification and playbook selection, ensuring that technical decisions remain predictable and auditable. The LLM is relegated to an advisory role, providing contextual analysis that is processed through a rigorous pipeline of validators, guardrails, and output sanitizers. This ensures that even if the model provides incomplete or hallucinated advice, the system defaults to safe, predefined operational policies. The architecture also includes a durable SQLite-based queue to prevent duplicate tasks and ensure recovery from interruptions without re-executing firewall rules.

Findings and Performance

Experiments conducted in a laboratory environment confirmed that the system maintains functional correctness and safety. The Rule Engine consistently matched predefined routing matrices, and the durable queue successfully processed 100 events without errors or duplicate actions. While the LLM provided useful advisory context, the study found that local inference was the primary processing bottleneck, adding significant latency compared to the deterministic baseline. The authors conclude that for well-defined alert categories, deterministic processing is superior, while LLMs should be reserved for complex, novel scenarios requiring deeper contextual synthesis.

Why It Matters

This research provides a blueprint for integrating generative AI into security operations without sacrificing control. By treating LLMs as untrusted advisory components rather than autonomous agents, organizations can leverage AI for analysis while maintaining the strict safety and auditability requirements necessary for critical network infrastructure.

Key Terms and Definitions

Static RAG — A retrieval method that uses exact category matching against a versioned, manual knowledge base instead of semantic vector-based search to ensure deterministic and reproducible results.
Durable Queue — A task management mechanism that preserves the state of incident processing in a database, allowing the system to recover from crashes without duplicating actions.
Guardrail — A validation component that inspects LLM output for consistency, authority, and structure, replacing invalid or unauthorized content with safe, deterministic defaults.
Response Engine — The isolated component responsible for executing firewall actions, which verifies whitelist status and incident state before applying any network-level changes.
Safe Fallback — A mechanism that ensures the system continues to function using deterministic rules if the LLM fails, times out, or provides unusable output.

Design and Evaluation of a Controlled Post-Alert Incident Orchestration and Response Subsystem Using a Rule Engine and a Local Large Language Model: Research Paper Podcast