AlphaWiSE: Adaptive Weight Interpolation for Continual Multimodal Representation Learning

Multimodal models like CLIP are typically trained on static datasets, but real-world applications require continual adaptation to new data. A major challenge is that sequential fine-tuning often disrupts the cross-modal alignment (e.g., audio-to-text or image-to-text) learned in earlier phases. Standard continual learning methods attempt to balance stability and plasticity, but they typically result in a single model checkpoint that forces a uniform trade-off across all retrieval directions. This paper asks whether it is possible to compose multiple, differently-optimized checkpoints post-hoc to achieve a better balance between retaining old knowledge and adapting to new information.

The authors propose AlphaWiSE, a post-hoc weight-space interpolation method. Instead of training a new model from scratch or using complex architectural modifications, AlphaWiSE takes two frozen checkpoints: one optimized for plasticity (standard sequential fine-tuning) and one optimized for stability (e.g., using EWC, LwF, or iCaRL). The method fits a scalar interpolation coefficient for every individual parameter tensor in the model. These coefficients are optimized using a small exemplar memory to minimize the retrieval loss across multiple modalities. Because the coefficients are applied to frozen weights, the resulting model maintains the exact architecture and parameter count of the source models, requiring zero additional inference-time computation.

Experiments on the AudioSet dataset demonstrate that AlphaWiSE consistently outperforms individual continual-learning baselines across various retrieval directions (audio-to-text, image-to-audio, and image-to-text). By allowing different layers and modality-specific components to draw different proportions from the source checkpoints, the fused model achieves a more flexible and robust representation. The results show that this tensor-wise interpolation is more effective than standard global weight-averaging methods, providing a practical way to improve multimodal retrieval performance in constrained-memory, continual-learning settings.