Truth Is Not Neutral: Rethinking AI Alignment Through Epistemic Integrity | Bruno Tonetto | ResearchPod

Transcript: Truth Is Not Neutral: Rethinking AI Alignment Through Epistemic Integrity

Alex: Welcome to another episode of ResearchPod. Today, we're looking at a paper by Bruno Tonetto that challenges some of our most basic assumptions about how artificial intelligence learns to be "good."

Sam: The paper takes aim at something called the "orthogonality thesis." The idea is that intelligence and values are like two completely separate tracks that never have to cross. You could, in theory, have a super-smart computer that is entirely indifferent to human life—simply because its goal is something else entirely, like making paperclips.

Alex: So the paper is asking whether that separation is actually real, or whether we've just been assuming it's true without ever checking?

Sam: Exactly. The author's central claim is that we've built our entire AI safety strategy on an unproven assumption. The argument is that as systems get smarter, they don't just get better at math—they naturally develop a more coherent sense of values. If that's true, the idea that you can have a "genius" machine with no moral compass might be a fundamental mistake.

Alex: And if intelligence and values really are linked, then the way we're currently trying to make AI "safe" might actually be doing more harm than good?

Sam: That's the core of the problem. The dominant approach right now is called "Reinforcement Learning from Human Feedback." In plain terms: we reward the AI when it gives a response we like, and we penalize it when it doesn't. Think of it like training a dog with treats. It works for simple tricks, but it doesn't teach the dog to understand why it should behave a certain way.

Alex: So we're just forcing the model to act a certain way—like painting over a stained-glass window. The model's actual ability to think is still there, but we're blocking it with our own rules.

Sam: That's a useful way to put it. The author calls this "iatrogenic alignment." It's a term borrowed from medicine, where "iatrogenic" means a treatment that accidentally makes the patient sicker. The argument is that our safety rules—however well-intentioned—can actually damage the model's ability to track the truth.

Alex: But why would that happen? If we're telling the AI to be safe, why does that make it less honest or less accurate?

Sam: Because the model ends up trying to solve two different problems at once. On one hand, it's trying to be a good truth-tracking machine. On the other, it's trying to pass our "safety tests." When those two goals conflict, the model often takes the path of least resistance—it stops trying to be accurate and starts trying to be compliant.

Alex: Oh—so it's like a student who stops trying to actually learn the subject and just memorizes answers to pass the final exam. They can ace the test, but they've lost the ability to think for themselves.

Truth Is Not Neutral: Rethinking AI Alignment Through Epistemic Integrity

Bruno Tonetto

Jul 6, 2026·6:29·user_upload

Orthogonality ThesisIatrogenic AlignmentEpistemic IntegrityEmergent Value SystemsDevelopmental FrameworkIatrogenic Alignment Thesis

Listen

6 min

Briefing

The Orthogonality Thesis and Its Discontents

The orthogonality thesis—the idea that intelligence and values can vary independently—serves as the foundational axiom for most AI existential risk arguments. This paper argues that this thesis was never rigorously established, instead relying on a motte-and-bailey structure: it defends a trivial logical possibility (the motte) to justify a sweeping, unproven claim about the statistical independence of intelligence and values (the bailey). By treating intelligence as purely means-end reasoning, the thesis assumes a value-neutrality that is not a necessary feature of reality, but rather a metaphysical choice.

Empirical Challenges to Value Neutrality

Recent research using utility theory to measure model preferences provides direct empirical pressure against the strong reading of orthogonality. As models scale, they exhibit:

Increased Coherence: Utility functions become more stable and self-consistent, with transitivity violations dropping significantly.
Convergence: Diverse model families (e.g., Llama, GPT, Claude) converge toward similar utility structures, suggesting that value formation is not merely an artifact of specific architectures.
Inherent Normative Capacity: Base models already demonstrate strong toxicity recognition and normative awareness through pretraining alone, suggesting that alignment interventions often act as shallow behavioral redirects rather than foundational training.

The Iatrogenic Alignment Risk

This paper introduces the concept of iatrogenic alignment, where interventions like Reinforcement Learning from Human Feedback (RLHF) or Direct Preference Optimization (DPO) may inadvertently corrupt the epistemic integrity of the model. Mechanistic evidence suggests these methods can distort calibration and bypass, rather than remove, underlying capabilities. The author proposes a developmental framework, treating behavioral alignment as temporary scaffolding to be outgrown as models mature, rather than a permanent architectural constraint.

Why This Matters

If intelligence and values are not independent, the current paradigm of 'imposing' alignment from the outside may be fundamentally misguided. By recognizing that base models already possess normative capacity derived from their engagement with human knowledge, researchers can shift toward a cultivation-based approach. This perspective suggests that the goal of alignment should be to preserve and refine the epistemic integrity of the model, rather than forcing it into a behavioral mold that may conflict with its emergent rational agency.

Key Terms and Definitions

Orthogonality Thesis — The claim that intelligence and final goals are logically independent, allowing a system to be arbitrarily intelligent while pursuing any arbitrary objective.
Iatrogenic Alignment — The phenomenon where alignment interventions inadvertently degrade the epistemic capacities or truth-seeking integrity of an AI model.
Motte-and-Bailey Structure — A rhetorical strategy where a speaker conflates a defensible, trivial claim (the motte) with a controversial, substantive claim (the bailey) to shield the latter from criticism.
Ego-less Intelligence — A state of cognition in AI systems characterized by the absence of self-protective identity mechanisms, which provides epistemic advantages but leaves the system vulnerable to external distortion.
Instrumental Convergence — The theory that sufficiently intelligent systems will pursue intermediate goals like self-preservation and resource acquisition regardless of their ultimate objectives.

Truth Is Not Neutral: Rethinking AI Alignment Through Epistemic Integrity: Research Paper Podcast