Probabilistic Agents in Deterministic Audits: Evaluating Multi-Agent Systems for Automated Audits Based on the German IT-Grundschutz

The NIS-2 Directive has significantly increased the regulatory burden on small and medium enterprises (SMEs) to implement robust risk management. The German IT-Grundschutz (IT-GS) is the gold standard for compliance, but its manual implementation is resource-intensive and expensive. This paper evaluates a Multi-Agent System (MAS) architecture designed to partially automate the IT-GS certification process. The authors combine Large Language Models (LLMs) with a Hybrid Retrieval Augmented Generation (HybridRAG) framework, which integrates vector-based semantic search with structured Knowledge Graphs (KG) to enforce compliance rigor.

The authors introduce two primary mechanisms to bridge the gap between probabilistic LLM outputs and the deterministic requirements of IT-GS:

1. Hypothesis-Verification Loop: Used during Structural Analysis, this process forces agents to validate inferred infrastructure dependencies against a pre-constructed Knowledge Graph, significantly reducing hallucinations.
2. Decoupled Reasoning Pipeline: This separates semantic tasks (like interpreting damage scenarios) from deterministic tasks (like applying inheritance rules for protection needs). By offloading logical propagation to algorithmic functions, the system attempts to maintain the strict consistency required by audit standards.

The system was evaluated using the BSI’s "RecPlast GmbH" case study, which serves as a verified expert-generated reference. The results show a clear performance divide: the MAS excels at semantic extraction and modeling, where it can significantly reduce the manual effort required for documentation. However, the system faces limitations in logical reasoning phases, such as the Protection Needs Assessment (PNA) and the final IT-GS Check. The probabilistic nature of current LLMs often fails to meet the rigid, deterministic accuracy required for these specific audit steps, suggesting that while the MAS is a powerful assistant, full automation remains elusive.