ResearchPod Summary
As generative AI tools become increasingly capable of drafting legal filings and judicial opinions, the legal system faces a fundamental challenge to its legitimacy. The author argues that a legal opinion is not merely a product of information transmission, but a record of a decisionmaker's authentic intellectual and emotional engagement with a case. When judges outsource the drafting of these opinions to AI, they risk abandoning the non-delegable duty to think, which is essential to the rule of law. This duty ensures that the person who decides a case is the same person who has heard and considered the arguments presented.
Large language models (LLMs) are fundamentally different from human decisionmakers. While they can produce fluent text, they lack a world model, intent, and the ability to experience the social and moral stakes of a legal dispute. The author highlights several failure modes, including the instability of LLM outputs, where small changes in prompts lead to divergent results, and the tendency of these models to hallucinate legal authorities. Furthermore, the opacity of these systems makes it impossible for the public or the parties involved to understand how a specific rationale was generated, thereby undermining the accountability required for judicial decisions.
Drawing on the philosophy of language, the author distinguishes between the designative-instrumental view of language—which treats words as mere tools for representing facts—and the expressive-intrinsic view, which sees language as a way to articulate and refine human understanding. Judicial reasoning belongs to the latter category. The process of writing an opinion is a form of thinking; it is through the effort of articulation that a judge clarifies their reasoning and confronts the complexity of the law. By delegating this process to a machine, the judge loses the opportunity for the critical reflection that constitutes the essence of wise judgment.
Alex: Welcome to another episode of ResearchPod. Today we're looking at Frank Pasquale's "The Non-Delegable Duty to Think." Sam, from the title alone, this sounds like a direct challenge to how courts and agencies are currently deploying AI.
Sam: It is, and the argument is more precise than it might first appear. Pasquale isn't claiming that AI outputs are necessarily wrong. He's claiming that legal legitimacy is constituted by the process of reason-giving, not just the quality of the final opinion. And because LLMs lack genuine agency, they can only simulate that process — which he argues is categorically insufficient for a legitimate legal system.
Alex: So the problem isn't accuracy. It's that the deliberative act itself is missing.
Sam: Exactly. He uses an analogy that lands well: using an LLM to draft a judicial opinion is like buying a finished painting and claiming you painted it. The artifact exists, the brushwork is there — but the intellectual process that gives it meaning is absent. Pasquale calls the duty to think a sine qua non for justice, something that can't be contracted out without the whole enterprise losing its foundation.
Alex: Where does he locate this duty? Is it a novel claim, or is he arguing it's already embedded in legal tradition?
Sam: The latter. He argues the duty is latent in existing doctrine — in the requirements for written opinions, in appellate review, in the expectation that a judge has actually grappled with the record. The novelty is recognizing that AI delegation quietly violates those expectations while leaving the formal structure intact.
Alex: Which is precisely what makes it hard to detect.
Sam: Right. And the mechanism he's most concerned about is what's already happening in administrative agencies — LLMs being used to draft decisions, often with Retrieval-Augmented Generation layered on top. RAG makes the output look highly authoritative: it pulls from real case law, cites real statutes, produces fluent legal prose. The risk is rubber-stamping — a human signs off on AI output to hit productivity targets, without having done the deliberative work the signature is supposed to certify.
Alex: So the document passes every formal check, but the cognitive process it's meant to represent never occurred.
Ultimately, the author contends that the legal system must explicitly recognize and preserve the duty of active mental consideration. This requires that judges not only review the final output but also personally grapple with the facts and arguments. While workload pressures are real, they do not justify an abdication of judicial responsibility. The paper calls for a legal culture that prioritizes thoughtful, human-centered adjudication over the efficiency of automated rationalization.
AI-generated third-party summary by ResearchPod. Not official content or an endorsement by the paper authors or affiliated organizations.
Sam: That's the core of it. And Pasquale is careful to say this isn't just a concern about errors slipping through. Even a hypothetically perfect LLM would fail his test, because the failure is structural. LLMs predict the next token based on statistical patterns across training data. They don't weigh competing moral considerations, they don't sit with the tension between precedent and equity, they don't experience the discomfort of a hard case. Those aren't peripheral features of legal reasoning — they're constitutive of it.
Alex: That's a claim a lot of AI researchers would push back on. How does Pasquale handle the objection that sufficiently sophisticated pattern-matching might be functionally indistinguishable from reasoning?
Sam: He doesn't fully resolve it, and that's a fair place to push. His response is essentially that legal legitimacy isn't purely a functional question — it's also a question of institutional design and democratic accountability. Even if we couldn't tell the difference from the outside, the accountability structure changes fundamentally. When a judge reasons through a case, there's a human being who can be questioned, who can be reversed on appeal, who bears responsibility. When a machine generates the reasoning, that chain of accountability becomes diffuse in ways the existing system isn't built to handle.
Alex: And if the accountability chain breaks down, what does he think follows?
Sam: He uses the phrase "automated social engineering" — a system where machines validate machines, where the outputs of one model become the training data or the precedent for the next, and human judgment gets progressively squeezed out. The concern isn't a single bad decision; it's a gradual drift where the rule of law becomes a rule of statistical inference, and no individual human is responsible for any of it.
Alex: That's a systemic risk argument, not just a philosophical one.
Sam: Precisely. And his prescriptive point follows from that. He's not calling for a ban on AI in legal contexts — he's calling for the duty to be made explicit before automation makes it practically impossible to enforce. Once courts and agencies are structurally dependent on LLM-generated reasoning, the political and institutional cost of reversing course becomes prohibitive. The window to establish the norm is now, not after the infrastructure is locked in.
Alex: So the paper is as much a warning about institutional path dependence as it is a philosophical argument about the nature of legal thought.
Sam: That's a fair reading. The philosophical argument about what reasoning is does the foundational work. But the urgency comes from the institutional claim: that the process of writing a legal opinion isn't just a vehicle for communicating a conclusion — it is a form of thinking, and delegating it doesn't just change how the work gets done, it changes what the work is.
Alex: That's a distinction worth sitting with. Thanks for walking us through it, Sam. And thanks to everyone listening — this has been ResearchPod.