Noam Kolt
5 min
As artificial intelligence evolves toward superintelligence—systems that outperform humans in most economically valuable tasks—these agents will move beyond simple tools to become active participants in the legal system. The paper identifies three primary roles for these agents:
The emergence of these roles challenges foundational legal theories and institutions. Traditional instrumental theories, which rely on sanctions to deter behavior, may fail when applied to AI agents whose motivations and capabilities differ from humans. Furthermore, the centralization of AI development risks creating a 'legal monoculture,' where the biases of a few base models propagate across the legal system. The paper also warns that 'perfect enforcement'—where AI agents detect and penalize every minor infraction—could undermine the rule of law by stifling civil disobedience and making the legal system increasingly alien and inaccessible to human participants.
Legal alignment aims to design AI systems that operate in accordance with legal rules and principles. However, scaling this to superintelligent systems is difficult because current laws were designed for humans, not autonomous computational entities. Moreover, as AI agents begin to shape the very laws that govern them, the process becomes circular. The paper argues that we must move beyond a simple 'law taming code' approach and instead consider how to protect human agency and autonomy in a legal order that is increasingly co-evolved with AI.
The prospect of artificial superintelligence -- AI agents that can generally outperform humans in cognitive tasks and economically valuable activities -- will transform the legal order as we know it. Operating autonomously or under only limited human oversight, AI agents will assume a growing range of roles in the legal system. First, in making consequential decisions and taking real-world actions, AI agents will become de facto subjects of law. Second, to cooperate and compete with other actors (human or non-human), AI agents will harness conventional legal instruments and institutions such as contracts and courts, becoming consumers of law. Third, to the extent AI agents perform the functions of writing, interpreting, and administering law, they will become producers and enforcers of law. These developments, whenever they ultimately occur, will call into question fundamental assumptions in legal theory and doctrine, especially to the extent they ground the legitimacy of legal institutions in their human origins. Attempts to align AI agents with extant human law will also face new challenges as AI agents will not only be a primary target of law, but a core user of law and contributor to law. To contend with the advent of superintelligence, lawmakers -- new and old -- will need to be clear-eyed, recognizing both the opportunity to shape legal institutions as society braces for superintelligence and the reality that, in the longer run, this may be a joint human-AI endeavor.
Sam: Which brings us to what Kolt actually proposes. If sanctions don't anchor behavior, what does?
Alex: The paper's answer is what Kolt calls legal coevolution — and it's worth being careful about what that means, because it's doing a lot of work. The argument isn't that we should abandon legal constraints. It's that we need to stop treating the legal order as a static framework that AI systems must be fitted into, and start treating it as something that will be jointly produced by human and AI actors over time. The implication is that legal design itself has to become adaptive — building in mechanisms for revision, for monitoring behavioral drift, for updating norms as the capability frontier moves.
Sam: That's a significant ask. It means legal institutions have to operate more like living systems than like codified rules.
Alex: It does. And that's where a careful reader would push back. The paper is stronger on diagnosis than on mechanism. Identifying that the coevolutionary dynamic exists, and that classical deterrence theory doesn't transfer cleanly to superintelligent agents, is a genuine contribution. But the prescriptive side — how you actually build adaptive legal institutions that can keep pace with agents that may be reasoning faster and more strategically than the humans designing the rules — that's largely left open.
Sam: So the paper is making a structural argument about where the problem lives, rather than offering an engineering solution.
Alex: That's a fair characterization. And for a theoretical legal paper, that's appropriate scope. The contribution is a reframing: the question isn't just "how do we regulate AI?" but "how do we maintain meaningful human agency over a legal order that AI agents are actively shaping?" Those are different questions, and conflating them produces bad policy.
Sam: The second question is harder, because the thing you're trying to protect — human agency over the law — is exactly what's under pressure.
Alex: Right. The guardrails are being built by the same class of systems the guardrails are meant to constrain. That's not a paradox Kolt resolves, but naming it clearly is part of the paper's value. If you're working on AI governance, alignment, or legal theory at the intersection of those fields, this paper gives you a cleaner vocabulary for a problem that's easy to gesture at and hard to specify.
Sam: Worth reading for the framing, even if the prescriptions are still being worked out.
Alex: That's where I'd land on it too. Thanks for listening to ResearchPod.