Practical Judgment, Virtue, and Intuition in the Use of Opaque AI-Enabled Systems

As AI systems increasingly rely on deep neural networks and large-scale machine learning, they often function as 'black boxes.' This opacity—where neither the user nor the developer can fully trace how inputs lead to specific outputs—creates significant operational, legal, and ethical risks. When these opaque systems are integrated into autonomous platforms, these risks are compounded because the systems may execute critical functions without real-time human oversight, potentially leading to unpredictable or harmful outcomes that cannot be easily reversed.

While many researchers advocate for Explainable AI (XAI) to solve the problem of opacity, the authors argue that XAI is often technically limited and insufficient for complex, real-world decision-making. Instead of attempting to force transparency onto inherently opaque algorithms, the authors suggest shifting the focus to the human operators. By leveraging human virtues—such as practical wisdom, restraint, and moral intuition—operators can bridge the gap between machine outputs and the ethical or legal requirements of a specific domain.

Drawing on Aristotelian virtue ethics, the paper posits that human expertise is defined by the ability to recognize salient features of a situation that cannot be codified into rules. In high-stakes environments like the military, a system might correctly identify a target according to its programming, but fail to perceive context-dependent moral factors that would lead a human to hold fire. The authors argue that training regimens should move away from purely rule-based compliance and toward cultivating the 'practical wisdom' that allows humans to act as a necessary, non-algorithmic check on autonomous systems.