Persistent Sparse Autoencoders: Learning Feature Timescales in Language Models

Standard Sparse Autoencoders (SAEs) decompose language model hidden states into sparse, interpretable features but treat each token independently. This atemporal approach fails to capture the semantic context—such as topic, style, or task state—that persists across a sequence. The authors introduce Persistent Sparse Autoencoders (Persistent SAEs), which augment standard SAEs by assigning a learned persistence coefficient to every feature. This allows the model to learn, through reconstruction alone, which features should act as fast, token-specific detectors and which should act as slow, persistent state variables.

The authors demonstrate that Persistent SAEs achieve competitive reconstruction quality compared to standard baselines while naturally organizing features by timescale. Most features are identified as 'fast' (low persistence), acting as locally interpretable detectors for specific tokens. A smaller subset of 'slow' (high persistence) features emerges, which effectively tracks topic-level semantic context. In experiments using MMLU sequences, these slow features form coherent subject-specific clusters that outperform standard SAEs and raw hidden states in subject-classification tasks. Furthermore, a case study on prompt-injection monitoring shows that these slow features preserve detection signals and remain causally effective over long contexts, suggesting they are well-suited for monitoring and steering model behavior.

This work addresses a fundamental limitation in current interpretability research: the inability of standard SAEs to explicitly represent the temporal evolution of information in language models. By enabling features to learn their own timescales, Persistent SAEs provide a more nuanced view of how LLMs maintain context. This approach offers a promising path for building more robust monitoring tools that can track semantic states over long sequences, potentially improving our ability to detect and mitigate adversarial inputs like prompt injections.