Author-updated Summary
Verified author edit
Mechanistic interpretability aims to open the black box of large language models (LLMs) by using interpretable replacement networks (IRNs)—such as sparse autoencoders or transcoders—to approximate dense internal computations with sparse, human-readable features. This paper investigates whether these IRNs are truly faithful to the underlying model, particularly when subjected to adversarial input perturbations that are semantically equivalent to clean inputs.
The authors evaluate the robustness of IRNs across five open-weight model families (GPT-2, Gemma 2, Gemma 3, Llama 3.2, and R1-Distill-Qwen). They first demonstrate that minor synonym-based perturbations can cause the dominant features of an IRN to flip, potentially misleading safety auditors. To address this, they introduce a formal verification framework that uses reachability analysis to compute a sound upper bound on the faithfulness gap between the IRN and the model. They further propose verification-aware training, which incorporates a reachable-set loss to align the IRN's manifold with the underlying model.
The study reveals that existing IRNs are highly unstable under adversarial attacks, with feature overlap dropping significantly as perturbations increase. The authors show that their formal verification framework can successfully bound the faithfulness gap, which tends to widen in deeper layers of the models. Crucially, they demonstrate that verification-aware training substantially tightens these certified bounds—reducing the faithfulness gap by approximately 90% in tested scenarios—and results in more sparse and interpretable feature representations.
This work provides the first formal guarantees for mechanistic interpretability, moving the field beyond empirical evaluations on clean data. By enabling safety auditors to certify the faithfulness of an interpretation, this framework helps ensure that the features read off an IRN actually reflect the underlying model's computation, rather than being artifacts of an approximate stand-in. This is a critical step toward making mechanistic interpretability a reliable tool for AI safety and auditing.
[[RP_SECTION:faithfulness-as-verification-problem|Faithfulness as verification problem]]
Sam: [measured, grounded] The faithfulness of an interpretability tool is itself a formal verification problem—not merely an empirical one. Interpretations are only as reliable as the provable bounds on their divergence from the underlying model. That's the central claim from a recent preprint by Tobias Ladner and Matthias Althoff at the Technical University of Munich.
Alex: [curious, analytical] So the standard way we check if an interpretability tool works—testing it on a handful of clean examples and seeing if the features look plausible—is fundamentally insufficient?
Sam: [steady, precise] That's exactly the problem. The authors show that even minor synonym substitutions can flip the dominant features of an Interpretable Replacement Network, or IRN. Their motivating example is pointed: a safety auditor blocks a prompt about culturing pathogens because the IRN highlights biology-related features. Paraphrase that same prompt to mention workplace culture, and the IRN's features shift to HR topics—the auditor approves the request. The underlying model hasn't changed. The interpretation has. That's not a minor calibration issue; it's the IRN failing to faithfully mirror the model's actual computation.
Alex: [processing] That's a significant vulnerability, especially for any deployment context where the interpretability tool is doing real gatekeeping work. How do they actually prove the tool is wrong? [[RP_SECTION:reachability-analysis-for-interpretabili|Reachability analysis for interpretability]]
Sam: [teaching mode, deliberate] They adapted reachability analysis—a technique from safety-critical control systems—to the latent spaces of these networks. The core construction is a difference network: you subtract the IRN's reconstruction from the model's actual layer output, and then ask how large that difference can get across all inputs within some perturbation set. They use zonotope-based reachability to bound that difference mathematically. If the bound is tight, the IRN is a faithful proxy. If it's loose, the interpretation is unverified and potentially misleading—and crucially, you now have a certificate telling you which.
Alex: [analytical] And the zonotope representation is doing real work there, right? It's what makes the bound computationally tractable rather than just a theoretical object.
Sam: [nodding in voice] Exactly. Zonotopes are closed under the affine operations that dominate these networks, so you can propagate the reachable set through layers without the representation exploding in complexity. The trade-off is that the bounds are conservative—they're valid upper bounds on the error, not tight ones—but for a safety auditing context, a conservative guarantee is still a guarantee.
AI-generated third-party summary by ResearchPod. Not official content or an endorsement by the paper authors or affiliated organizations.
Alex: [leaning in] So they've diagnosed the problem formally. Did they also try to fix it? [[RP_SECTION:verification-aware-training-methods|Verification-aware training methods]]
Sam: [calm] They did. They introduced verification-aware training, which adds a reachable-set volume loss to the training objective. The intuition is straightforward: during training, you're not just minimizing reconstruction error on observed inputs, you're penalizing the size of the reachable error set under perturbation. That forces the IRN to stay closer to the model's manifold even under adversarial pressure—not just on the clean examples you happened to test.
Alex: [thoughtful] And does it actually work? Tightening a bound during training could easily just be overfitting to the perturbation structure you chose.
Sam: [measured] The reported results are encouraging. The verified upper bound on the error set drops by roughly ninety percent compared to a standard IRN trained without the reachability objective. That's the load-bearing number in the paper. The authors are appropriately careful not to claim the underlying models are now robust—they're not—but the interpretability tool itself becomes a much tighter proxy for what the model is actually doing.
Alex: [processing] Which raises the deeper question: how much of the faithfulness gap is a property of the IRN architecture, and how much is baked into the model it's trying to interpret? [[RP_SECTION:limitations-and-model-architecture|Limitations and model architecture]]
Sam: [steady] Both contribute, and the authors are honest about that. As you move into deeper layers, the model's computations involve non-linearities that simple auxiliary networks genuinely struggle to reconstruct. Polysemanticity compounds this—when a single neuron responds to multiple unrelated concepts, there's no clean interpretation to recover, and any IRN is going to be approximating something underspecified. The formal verification framework doesn't dissolve that problem. What it does is tell you, precisely, how much drift you're tolerating. You're not pretending the tool is perfect; you're certifying the error budget.
Alex: [analytical] That's a meaningful reframe. Instead of asking "does this interpretation look right," you're asking "what's the worst-case gap between this interpretation and the model, and is that gap acceptable for this use case." [[RP_SECTION:interpretability-as-co-design|Interpretability as co-design]]
Sam: [measured, concluding] Precisely. And that reframe opens a path toward compositional auditing—in principle, you could stack layer-wise guarantees and propagate a bound through the entire network. Current bounds are conservative enough that this isn't yet practical at scale, but it establishes the formal scaffolding. The more consequential implication the authors gesture toward is architectural: if you want interpretability tools that are verifiable by construction, you probably need to design models with that in mind from the start, rather than retrofitting verification onto architectures that were never built for it. That's a significant constraint on the design space, but it's the honest conclusion if you take provably safe auditing seriously.
Alex: [quiet, reflective] It reframes the whole enterprise. Interpretability isn't just a post-hoc analysis problem—it's a co-design problem.
Sam: [grounded] That's the implication. And the contribution here is less about the specific numbers and more about establishing that the question is even well-posed. You can ask for a formal certificate on an interpretability tool. You can train toward tighter certificates. Whether the field takes that seriously is a separate question, but the technical foundation is now there. Thanks for listening to ResearchPod.