The Art of Not Forgetting

Catastrophic forgetting is typically addressed as an external training-time problem, such as using replay buffers or regularization (e.g., EWC). The authors hypothesize that forgetting is instead a structural consequence of how global backpropagation assigns credit. They propose the Cognitive Memory Primitive (CMP) architecture, which replaces backpropagation with local, sparse, and gradient-free learning rules to see if this structural change inherently mitigates forgetting.

CMP is built from several established components: sparse relational binding, a two-tier competitive memory (inspired by Kanerva’s Sparse Distributed Memory), a predictive-coding hierarchy, and a local delta-rule readout. The authors introduce a novel 'weight-protect' mechanism that uses raw weight-movement magnitude as a gradient-free importance signal to throttle plasticity. They test this architecture on a domain-incremental protocol across 15 diverse text domains, comparing it against a parameter-matched Transformer baseline using online EWC.

CMP significantly outperforms the Transformer baseline in backward transfer (BWT) metrics, showing 15-19x less forgetting across the 15-domain sequence. This result remains consistent across different random seeds and domain-order controls. However, the authors are transparent about the model's limitations: CMP suffers from a substantial accuracy gap compared to Transformers, fails to show improvements on vision tasks (Split-MNIST), and does not currently integrate well with other high-accuracy local learning mechanisms. The authors emphasize that while the architecture succeeds in its narrow goal of resisting forgetting, it is not a general-purpose replacement for attention-based models.

This work challenges the prevailing view that catastrophic forgetting is merely a defect to be patched. By demonstrating that a system can be designed to resist forgetting through local, sparse architectural constraints, the authors provide a falsifiable, biologically-inspired alternative to global backpropagation for continual learning scenarios.