MJEPA: A Simple and Scalable Joint-Embedding Predictive Architecture for Audio-Visual Learning

How can we effectively learn joint audio-visual representations without the complexity of modality-specific encoders, contrastive losses, or heavy data augmentations? The authors investigate whether a single, unified encoder can learn from both modalities simultaneously using a simple predictive objective.

The authors introduce MJEPA (Multimodal Joint-Embedding Predictive Architecture). Unlike existing methods that use separate encoders for audio and video, MJEPA uses a single shared encoder and a shared predictor. The model is trained using a Joint-Embedding Predictive Architecture (JEPA) objective, which predicts masked representations in a latent space. Crucially, the authors implement both intra-modal prediction (predicting masked regions within a modality) and cross-modal prediction (predicting one modality from the other). This cross-modal alignment is the key mechanism that allows the model to benefit from the synergy between audio and video.

The study reveals that a shared encoder without explicit cross-modal alignment actually performs worse than unimodal baselines, as the model struggles to reconcile the different statistical properties of audio and video. However, by adding cross-modal prediction, the model achieves positive transfer, where each modality improves the other. The resulting frozen ViT-g model outperforms prior frozen baselines on AudioSet-20K by over 6.8 mAP and surpasses fully fine-tuned models on ESC-50 and FSD50K. Furthermore, the model remains competitive on video benchmarks while using significantly less training data than previous approaches.

This work simplifies the architecture of multimodal learning, moving away from complex, multi-component systems toward a unified, scalable design. By proving that a single encoder can effectively handle multiple modalities, the paper provides a blueprint for building more efficient and generalizable foundation models that can process heterogeneous data streams without needing specialized, modality-specific pipelines.