Engineering Trustworthy Agentic AI for Critical Systems

Research Overview

This paper addresses the critical gap between the rapid development of agentic AI—systems capable of autonomous perception, planning, and tool use—and the rigorous requirements of safety-critical engineering. While current literature often evaluates agents based on task performance, the authors argue that for deployment in high-stakes environments like power grids, robotics, and communication networks, trustworthiness must be treated as a fundamental engineering property. The study provides a structured, trust-centered analysis of agentic architectures, mapping them onto a comprehensive assurance workflow.

The Trustworthiness Model

The authors introduce a five-dimensional model for evaluating agentic systems: safety and constraint satisfaction, robustness and reliability, transparency and interpretability, accountability and auditability, and privacy and security. By mapping these dimensions onto a ten-point agentic assurance workflow—spanning from initial perception to final audit—the authors identify recurring design patterns and failure modes. They argue that agentic AI trustworthiness is not a collection of disparate problems but a single, cross-domain challenge that requires a unified, graded certification regime similar to those used in aviation or automotive engineering.

Engineering Domains and Synthesis

The paper examines these principles across four constraint-bound domains: power systems, autonomous vehicles/robotics, high-performance computing, and communication networks. By comparing these fields, the authors reveal that while domain-specific vocabulary differs, the underlying risks—such as the lack of domain-aware safety guarantees under distribution shift and the absence of standardized evaluation—are structurally identical. The study concludes by outlining a path toward reusable assurance infrastructure, emphasizing that the next phase of AI development must prioritize verifiable safety over architectural novelty.