Beyond the Hard Budget: Sparsity Regularizers for More Interpretable Top-k Sparse Autoencoders

Research Context and Motivation

Sparse autoencoders (SAEs) are essential for interpreting the internal representations of vision foundation models (VFMs) by decomposing complex, polysemantic activations into more interpretable, monosemantic features. The standard Top-k SAE enforces sparsity by retaining only the k most active latent units per input, a design choice intended to avoid the feature shrinkage and dead-neuron issues associated with traditional L1-regularized SAEs. However, this hard constraint is rigid; it forces a fixed number of active units regardless of the input's actual complexity and makes the model prone to overfitting to the specific value of k used during training.

Proposed Regularization Approach

The authors introduce two sparsity regularizers that act on the latent activations before the Top-k selection, restricted to units that are active within a given batch. The first, an off-support L1 penalty, targets the activations of units that are not selected by the Top-k operator, driving their sub-threshold responses toward zero to sharpen unit selectivity. The second, an L1/L2-ratio penalty, is a scale-invariant measure that encourages the model to concentrate its reconstruction on fewer effective latent units. By applying these penalties only to batch-active units, the authors avoid exacerbating the problem of dead neurons.

Empirical Findings

Across three vision foundation models (CLIP, SigLIP2, and a supervised ViT-L/16) and two datasets (ImageNet-1K and Open Images V7), the authors show that both regularizers consistently improve monosemanticity and class purity. The off-support L1 penalty generally yields the largest gains in interpretability, while the L1/L2-ratio penalty provides added robustness to the inference-time choice of k. The results demonstrate that hard architectural sparsity and soft sparsity regularization are not mutually exclusive but rather complementary, allowing for more flexible and interpretable latent representations.

This work provides a practical path to improving the interpretability of vision models without the need for complex architectural changes. By simply augmenting the training objective of existing Top-k SAEs, researchers can extract more coherent, concept-specific features. This is particularly valuable for auditing foundation models, as it allows for a more reliable mapping between latent activations and human-understandable visual concepts.