Sparse Autoencoders Encode Both Concepts and Functions: The Downstream Geometry of Feature Effects

Sparse autoencoders (SAEs) are widely used to decompose model activations into interpretable features. However, a significant gap remains between identifying a feature's meaning and understanding its causal impact on model behavior. The authors investigate whether SAE features act as reliable, reusable 'steering' vectors. To do this, they introduce Feature-Effect Geometry Analysis (FEGA), an unsupervised framework that measures the geometry of logit changes caused by ablating the same SAE feature across different contexts. By comparing these logit-space effects, the authors categorize features based on their functional roles: value-like (associated with static, factual information) and pointer-like (associated with context-dependent, operational tasks).

Across multiple SAE architectures—including ReLU, TopK, and Matryoshka Batch TopK—the authors find that consistent, one-dimensional steering is the exception rather than the rule. Most features do not behave as simple, reusable directions. Instead, the downstream effects of a feature form a 'cloud' in the logit space.

Pointer-like features, which support operations like copying or rule-following, exhibit predominantly diffuse effects. This suggests that even when these features are causally important for a task, they do not provide a stable, single-vector intervention. Value-like features, such as those representing factual attributes, show more structured, low-dimensional geometry, though these effects typically span multiple output directions rather than collapsing into a single steering vector. These results imply that a feature can be highly interpretable and causally relevant without being a reliable tool for steering model outputs.

This paper challenges the common assumption in mechanistic interpretability that a feature with a clear semantic label is necessarily a stable control variable. By shifting the focus from activation-based descriptions to the geometry of downstream effects, the authors provide a more rigorous diagnostic for evaluating SAEs. This work suggests that researchers should distinguish between what a feature detects and how its causal influence propagates, as the latter is often far more complex and context-dependent than previously assumed.