Sparsity, Superposition, and Forgetting: A Mechanistic Study of Representation Retention in Continual Learning

Research Approach

This paper investigates the mechanistic drivers of catastrophic forgetting in continual learning (CL) by moving away from complex, real-world datasets toward a controlled, synthetic toy-world. The authors introduce a generator-separator pipeline that allows them to define ground-truth latent features and tune variables like sparsity and directional overlap (superposition). By using a replay-based training protocol, they track how individual feature representations evolve over time, using Sparse Identification of Nonlinear Dynamical Systems (SINDy) to model the relationship between representation strength, superposition, and retention.

Key Findings

The authors present three primary insights into the mechanics of forgetting. First, they find that superposition is not a steady, uniform drift but rather a dynamic process that increases over time, punctuated by transient dips at task boundaries. This suggests that interference is most acute during the transition between tasks. Second, while higher feature sparsity forces the model to encode more information via superposition, this does not automatically cause forgetting. Instead, the authors demonstrate that representation strength acts as a critical buffer; features that are strongly represented are far more resistant to being overwritten, even when they share latent directions with other features. Finally, the study shows that tasks composed of sparser signals occupy more latent capacity, as measured by higher effective rank, indicating that the model allocates broader representational volume to sparse regimes.

These findings challenge the common intuition that superposition is inherently detrimental to knowledge retention. By showing that overlap interacts with representation strength and capacity allocation, the authors provide a more nuanced understanding of how neural networks manage limited resources in non-stationary environments. The proposed diagnostic tools and falsifiable hypotheses offer a principled way to evaluate CL algorithms, potentially guiding the design of more robust models that can better manage the trade-off between efficient compression and long-term memory.