From Geometric Recovery to Causal Validation: A Reproducible Audit of Sparse Autoencoder Features, from Superposition Geometry to Causal Inertness

Sparse autoencoders (SAEs) are the primary tool for decomposing neural network activations into interpretable features. The standard approach for evaluating these features is correlational: researchers measure the cosine similarity between a learned decoder atom and a known ground-truth direction. This paper argues that this metric is insufficient because it conflates two distinct processes: the geometric alignment of the decoder and the actual activation behavior of the encoder. By using a fully controlled synthetic model where ground truth is known, the author demonstrates that high cosine similarity does not guarantee that a feature is causally active.

The central contribution is a causal audit of recovered features using ablation and steering interventions. The study reveals that a significant portion of features—up to 77% in degraded SAEs and 9% in well-trained ones—are causally inert. These features pass the standard correlational recovery bar (cosine ≥ 0.90) but fail to influence the model's output when the feature is present. The author classifies this inertness into two categories: structural inertness, which arises from the geometry of antipodal-pair superposition and persists even in well-trained models, and competitive inertness, which is a pathology of TopK selection in degraded dictionaries.

The author provides a deterministic, model-agnostic tool called sae-causal-audit to facilitate these interventions. Beyond the synthetic experiments, the paper applies this audit to a production SAE (GPT-2-small). The results confirm that the causal inertness pattern persists in real-world models (14% inert). Furthermore, the audit identifies an "atom-collision" signal, where a small number of decoder atoms are repeatedly matched to dozens of semantically unrelated concepts, suggesting that current dictionary learning methods may be under-splitting features in ways that standard correlational metrics fail to detect.