Tobias Ladner, Matthias Althoff
6 min
Mechanistic interpretability aims to open the black box of large language models (LLMs) by using interpretable replacement networks (IRNs)—such as sparse autoencoders or transcoders—to approximate dense internal computations with sparse, human-readable features. This paper investigates whether these IRNs are truly faithful to the underlying model, particularly when subjected to adversarial input perturbations that are semantically equivalent to clean inputs.
The authors evaluate the robustness of IRNs across five open-weight model families (GPT-2, Gemma 2, Gemma 3, Llama 3.2, and R1-Distill-Qwen). They first demonstrate that minor synonym-based perturbations can cause the dominant features of an IRN to flip, potentially misleading safety auditors. To address this, they introduce a formal verification framework that uses reachability analysis to compute a sound upper bound on the faithfulness gap between the IRN and the model. They further propose verification-aware training, which incorporates a reachable-set loss to align the IRN's manifold with the underlying model.
The study reveals that existing IRNs are highly unstable under adversarial attacks, with feature overlap dropping significantly as perturbations increase. The authors show that their formal verification framework can successfully bound the faithfulness gap, which tends to widen in deeper layers of the models. Crucially, they demonstrate that verification-aware training substantially tightens these certified bounds—reducing the faithfulness gap by approximately 90% in tested scenarios—and results in more sparse and interpretable feature representations.
This work provides the first formal guarantees for mechanistic interpretability, moving the field beyond empirical evaluations on clean data. By enabling safety auditors to certify the faithfulness of an interpretation, this framework helps ensure that the features read off an IRN actually reflect the underlying model's computation, rather than being artifacts of an approximate stand-in. This is a critical step toward making mechanistic interpretability a reliable tool for AI safety and auditing.
Mechanistic interpretability has become the dominant lens for understanding frontier language models, as their inner workings are complex and inherently black boxes. To gain insights into these models, interpretable replacement networks (IRNs) are trained at all layers, exposing interpretable features through sparsely activated neurons. However, the faithfulness of an IRN is usually evaluated only empirically on clean data, and we show that even semantically minor input perturbations flip the dominant IRN features-and thus the human-understandable interpretation-across five open-weight model families (GPT-2 small, Gemma 2 2B, Gemma 3 1B, Llama 3.2 1B, R1-Distill-Qwen 1.5B). We propose the first formal verification framework for the faithfulness of an IRN, where reachability analysis certifies a sound upper bound of the faithfulness gap in adversarial scenarios. Moreover, we show that verification-aware training of IRNs substantially tightens this certified bound, restoring a feature-level interpretation that safety auditors can act on. Together, these results give, to the best of our knowledge, the first formal guarantees for mechanistic interpretability of large language models.
Alex: [leaning in] So they've diagnosed the problem formally. Did they also try to fix it? [[RP_SECTION:verification-aware-training-methods|Verification-aware training methods]]
Sam: [calm] They did. They introduced verification-aware training, which adds a reachable-set volume loss to the training objective. The intuition is straightforward: during training, you're not just minimizing reconstruction error on observed inputs, you're penalizing the size of the reachable error set under perturbation. That forces the IRN to stay closer to the model's manifold even under adversarial pressure—not just on the clean examples you happened to test.
Alex: [thoughtful] And does it actually work? Tightening a bound during training could easily just be overfitting to the perturbation structure you chose.
Sam: [measured] The reported results are encouraging. The verified upper bound on the error set drops by roughly ninety percent compared to a standard IRN trained without the reachability objective. That's the load-bearing number in the paper. The authors are appropriately careful not to claim the underlying models are now robust—they're not—but the interpretability tool itself becomes a much tighter proxy for what the model is actually doing.
Alex: [processing] Which raises the deeper question: how much of the faithfulness gap is a property of the IRN architecture, and how much is baked into the model it's trying to interpret? [[RP_SECTION:limitations-and-model-architecture|Limitations and model architecture]]
Sam: [steady] Both contribute, and the authors are honest about that. As you move into deeper layers, the model's computations involve non-linearities that simple auxiliary networks genuinely struggle to reconstruct. Polysemanticity compounds this—when a single neuron responds to multiple unrelated concepts, there's no clean interpretation to recover, and any IRN is going to be approximating something underspecified. The formal verification framework doesn't dissolve that problem. What it does is tell you, precisely, how much drift you're tolerating. You're not pretending the tool is perfect; you're certifying the error budget.
Alex: [analytical] That's a meaningful reframe. Instead of asking "does this interpretation look right," you're asking "what's the worst-case gap between this interpretation and the model, and is that gap acceptable for this use case." [[RP_SECTION:interpretability-as-co-design|Interpretability as co-design]]
Sam: [measured, concluding] Precisely. And that reframe opens a path toward compositional auditing—in principle, you could stack layer-wise guarantees and propagate a bound through the entire network. Current bounds are conservative enough that this isn't yet practical at scale, but it establishes the formal scaffolding. The more consequential implication the authors gesture toward is architectural: if you want interpretability tools that are verifiable by construction, you probably need to design models with that in mind from the start, rather than retrofitting verification onto architectures that were never built for it. That's a significant constraint on the design space, but it's the honest conclusion if you take provably safe auditing seriously.
Alex: [quiet, reflective] It reframes the whole enterprise. Interpretability isn't just a post-hoc analysis problem—it's a co-design problem.
Sam: [grounded] That's the implication. And the contribution here is less about the specific numbers and more about establishing that the question is even well-posed. You can ask for a formal certificate on an interpretability tool. You can train toward tighter certificates. Whether the field takes that seriously is a separate question, but the technical foundation is now there. Thanks for listening to ResearchPod.