Attention-Spectrum Regularization for Replay-Free Continual Multimodal LLMs

As multimodal large language models (MLLMs) are adapted to continuous streams of new visual domains and instructions, they frequently suffer from catastrophic forgetting, where performance on previously learned tasks degrades. Existing solutions often rely on replaying past data, storing pseudo-examples, or complex parameter allocation. This paper investigates whether the internal cross-modal attention structure—which dictates how a model focuses on visual evidence to answer specific questions—can be preserved to maintain skill-level performance without needing past data.

The authors propose Attention-Spectrum Regularization (ASR), a replay-free framework that treats cross-modal attention maps as two-dimensional signals. By applying a Fourier transform, the model extracts spectral statistics (scale and directional properties) that summarize how and where the model attends to visual features. ASR maintains a compact memory of skill-conditioned prototypes—Gaussian distributions representing these spectral signatures for different reasoning skills (e.g., counting, OCR, spatial reasoning). During continual fine-tuning, a phase-invariant spectral regularizer constrains the drift of these attention patterns, ensuring that the model retains its ability to ground visual evidence for old skills while allowing for adaptation to new tasks.

Exploratory analysis revealed that multimodal forgetting is strongly correlated with the drift of attention spectra and the loss of high-frequency visual focus. ASR effectively stabilizes these patterns without storing past images or questions. Experiments across multiple benchmarks, including VQA v2, VQACL, and various multimodal instruction-tuning tasks, demonstrate that ASR consistently reduces forgetting and improves final performance compared to strong baselines like replay-based methods and adapter-based tuning. The results suggest that preserving the internal structure of visual attention is a lightweight and highly effective mechanism for long-term multimodal learning.