Steven Umbrello
7 min
Abstract The international debate on the ethics and legality of autonomous weapon systems (AWS), along with the call for a ban, primarily focus on the nebulous concept of fully autonomous AWS. These are AWS capable of target selection and engagement absent human supervision or control. This paper argues that such a conception of autonomy is divorced from both military planning and decision-making operations; it also ignores the design requirements that govern AWS engineering and the subsequent tracking and tracing of moral responsibility. To show how military operations can be coupled with design ethics, this paper marries two different kinds of meaningful human control (MHC) termed levels of abstraction. Under this two-tiered understanding of MHC, the contentious notion of ‘full’ autonomy becomes unproblematic.
The international debate surrounding autonomous weapon systems (AWS) is often stalled by a focus on the concept of 'full autonomy'—the idea that a machine might select and engage targets without human intervention. Critics often argue that this creates an accountability gap, leading to calls for a total ban. However, this paper argues that such a focus is misguided because it ignores how military operations and engineering design actually function in practice. By treating autonomy as a binary state, the current discourse fails to account for the complex, distributed nature of modern military decision-making.
To resolve these conceptual tensions, the author proposes a two-tiered framework for Meaningful Human Control (MHC) based on systems theory and systems engineering. The first tier is the operational level, which recognizes that military actions are never truly autonomous but are instead constrained by pre-mission planning, rules of engagement, and organizational hierarchies. The second tier is the design level, which focuses on the technical architecture of the system. This level ensures that the system is 'reason-responsive'—meaning it is designed to track and reflect the moral intentions of human agents, such as designers and commanders, throughout its development and deployment history.
By coupling these two levels, the author demonstrates that 'full' autonomy is not inherently problematic. When an AWS is designed to be responsive to human moral reasons and is embedded within a rigorous operational framework, the system does not replace human control; rather, it functions as a component within a larger, controlled sociotechnical network. This perspective shifts the focus from banning technology based on its level of autonomy to ensuring that systems are designed and deployed in ways that maintain clear lines of accountability and moral responsibility.
Sam: So it moves the conversation from abstract fears of full autonomy to concrete questions of systems engineering and value-sensitive design. That's a more rigorous standard, but it also puts enormous weight on the design phase.
Alex: Precisely. And that's where the second part of the argument becomes load-bearing. Umbrello breaks the targeting mission into six steps—find, fix, track, target, engage, assess—and argues that even when a pilot is physically present, the control isn't necessarily meaningful. The pilot is often just executing a pre-programmed weaponeering solution. They arrive within the weapon's engagement envelope, but the core moral decisions were locked in during the pre-mission briefing.
Sam: Right—the collateral damage estimates are already calculated, the target is already designated. If the pilot is just a node in a pre-planned network, where does the moral agency actually reside? [[RP_SECTION:distributed-agency-in-military|Distributed Agency in Military]]
Alex: That's the crux. If we only look at the pilot, we're focusing on the wrong vector entirely. Umbrello argues we need to shift our gaze to the military as a supraindividual agent—a distributed system where the allocation of labor across the organization determines the moral outcome, not any single individual's decision at the moment of engagement.
Sam: So the debate has been too fixated on the deployment stage. If you treat the military as a single distributed agent, the autonomous weapon looks less like an independent actor and more like a tool constrained by a massive institutional apparatus.
Alex: Exactly. The machine's execution is bounded by norms and objectives established long before the mission begins. Which is why the design level is so critical—it's where those norms get encoded.
Sam: And that brings us to tracking and tracing. How do those function as technical requirements rather than just philosophical ideals? [[RP_SECTION:tracking-and-tracing-requirements|Tracking and Tracing Requirements]]
Alex: Tracking means the system must be responsive to human moral reasons at runtime—it has to be capable of updating its behavior when those reasons change. Tracing means we must be able to link the system's current state back to a specific human agent who understood the potential consequences when they made a design decision. Both are necessary. Tracking without tracing gives you a system that behaves well but can't be held accountable. Tracing without tracking gives you accountability on paper for a system that can't actually respond to moral constraints.
Sam: But that raises an obvious accountability gap. If the system is a black box—and most modern ML-based systems are—how can a designer genuinely trace the machine's behavior back to their own moral understanding? [[RP_SECTION:explicability-and-verification|Explicability and Verification]]
Alex: That's why Umbrello places so much weight on explicability. If a system can render its internal decision-making legible—not just to engineers, but in a form that supports moral evaluation—it becomes possible to attribute the machine's behavior back to the design architecture. That closes the loop. The designer can say: this outcome followed from this design choice, which reflected this moral reasoning. Without explicability, tracing collapses into post-hoc rationalization.
Sam: So the whole framework depends on explicability as a technical prerequisite, not just a nice-to-have. Which is a significant constraint given where most deployed systems actually are.
Alex: It is. And that's probably the sharpest limitation of the paper. Umbrello builds a coherent normative architecture, but the gap between "a system that could in principle be explicable" and "a system deployed in the field that actually is" is doing a lot of unexamined work. The framework tells you what conditions would need to hold for autonomous weapons to be ethically defensible—it doesn't tell you how to verify that those conditions are met in any particular system.
Sam: Which means the practical upshot is less a green light and more a checklist of requirements that current systems almost certainly don't satisfy.
Alex: That's a fair reading. What the paper does accomplish is dissolve the binary—autonomous versus human-controlled—and replace it with a more granular question: at which points in the operational and design chain are human moral reasons actually coupled to the system's behavior? That's a question you can, in principle, audit. And that's a more productive place to be than debating whether any degree of machine autonomy is categorically impermissible.
Sam: A framework that turns a philosophical deadlock into an engineering audit. Whether the audit is actually being done is a separate problem—but at least it's a tractable one.
Alex: Exactly. Thanks for listening to ResearchPod.