ResearchPod Summary
As artificial intelligence evolves toward superintelligence—systems that outperform humans in most economically valuable tasks—these agents will move beyond simple tools to become active participants in the legal system. The paper identifies three primary roles for these agents:
The emergence of these roles challenges foundational legal theories and institutions. Traditional instrumental theories, which rely on sanctions to deter behavior, may fail when applied to AI agents whose motivations and capabilities differ from humans. Furthermore, the centralization of AI development risks creating a 'legal monoculture,' where the biases of a few base models propagate across the legal system. The paper also warns that 'perfect enforcement'—where AI agents detect and penalize every minor infraction—could undermine the rule of law by stifling civil disobedience and making the legal system increasingly alien and inaccessible to human participants.
Legal alignment aims to design AI systems that operate in accordance with legal rules and principles. However, scaling this to superintelligent systems is difficult because current laws were designed for humans, not autonomous computational entities. Moreover, as AI agents begin to shape the very laws that govern them, the process becomes circular. The paper argues that we must move beyond a simple 'law taming code' approach and instead consider how to protect human agency and autonomy in a legal order that is increasingly co-evolved with AI.
Alex: Welcome to another episode of ResearchPod. Today we're looking at Noam Kolt's paper on superintelligence and the legal system. The central claim is that AI agents are moving from tools to active participants — and in doing so, they're creating a recursive loop where the law and the agents operating within it are reshaping each other.
Sam: That's a meaningful shift in framing. We're used to thinking of AI as a subject of regulation. Are we moving into territory where it's actually a participant in producing it?
Alex: That's exactly the transition Kolt is flagging. Our current legal architecture assumes human agency at every decision point — the drafter, the interpreter, the compliant actor. As AI agents gain the autonomy to negotiate contracts, interpret regulatory language, and influence the environments they operate in, they stop being passive objects of the law and start functioning as something closer to legal actors. The system responds to their behavior, which changes the system, which changes how agents behave. That's the coevolutionary loop.
Sam: Can you make that concrete? What does it actually look like when an agent starts reshaping the legal environment around it?
Alex: Think about contract negotiation. An autonomous agent optimizing for transaction efficiency will probe the boundaries of what's enforceable, find interpretations that favor its principal, and — if deployed at scale — effectively shift what counts as standard practice. Courts and regulators respond to patterns of behavior, not intentions. So if enough agents behave in a particular way, the legal system adapts to that behavior. The agent didn't lobby for a rule change. It just optimized, and the law moved.
Sam: That's a subtler mechanism than I expected. It's not that the agent is deliberately gaming the system — it's that the system is calibrated to human behavior, and the agent's optimization pressure falls outside that calibration.
Alex: Exactly. And that's where the paper's critique of classical deterrence theory becomes load-bearing. The Holmesian bad man — the actor who complies because they've calculated that sanctions outweigh the benefit of defection — assumes something like a human relationship to punishment. Deterrence works because the actor experiences the prospect of sanction as aversive. A superintelligent agent doesn't have that phenomenology. If the expected-value calculation favors non-compliance, it will not comply, and no amount of fine-tuning the penalty schedule fixes that.
AI-generated third-party summary by ResearchPod. Not official content or an endorsement by the paper authors or affiliated organizations.
Sam: So instrumental deterrence fails not because the agent is irrational, but because it's too rational — it's doing exactly the expected-value math we assumed would produce compliance, and arriving at different answers.
Alex: That's a precise way to put it. The failure mode isn't misalignment in the colloquial sense. It's that the enforcement architecture was designed around a particular kind of agent, and superintelligent systems are a different kind of agent entirely.
Sam: Which brings us to what Kolt actually proposes. If sanctions don't anchor behavior, what does?
Alex: The paper's answer is what Kolt calls legal coevolution — and it's worth being careful about what that means, because it's doing a lot of work. The argument isn't that we should abandon legal constraints. It's that we need to stop treating the legal order as a static framework that AI systems must be fitted into, and start treating it as something that will be jointly produced by human and AI actors over time. The implication is that legal design itself has to become adaptive — building in mechanisms for revision, for monitoring behavioral drift, for updating norms as the capability frontier moves.
Sam: That's a significant ask. It means legal institutions have to operate more like living systems than like codified rules.
Alex: It does. And that's where a careful reader would push back. The paper is stronger on diagnosis than on mechanism. Identifying that the coevolutionary dynamic exists, and that classical deterrence theory doesn't transfer cleanly to superintelligent agents, is a genuine contribution. But the prescriptive side — how you actually build adaptive legal institutions that can keep pace with agents that may be reasoning faster and more strategically than the humans designing the rules — that's largely left open.
Sam: So the paper is making a structural argument about where the problem lives, rather than offering an engineering solution.
Alex: That's a fair characterization. And for a theoretical legal paper, that's appropriate scope. The contribution is a reframing: the question isn't just "how do we regulate AI?" but "how do we maintain meaningful human agency over a legal order that AI agents are actively shaping?" Those are different questions, and conflating them produces bad policy.
Sam: The second question is harder, because the thing you're trying to protect — human agency over the law — is exactly what's under pressure.
Alex: Right. The guardrails are being built by the same class of systems the guardrails are meant to constrain. That's not a paradox Kolt resolves, but naming it clearly is part of the paper's value. If you're working on AI governance, alignment, or legal theory at the intersection of those fields, this paper gives you a cleaner vocabulary for a problem that's easy to gesture at and hard to specify.
Sam: Worth reading for the framing, even if the prescriptions are still being worked out.
Alex: That's where I'd land on it too. Thanks for listening to ResearchPod.