On the modality gap and the contrastive loss in multi-modal representation learning

Contrastive learning models like CLIP use independent encoders for different modalities (e.g., images and text) trained via the InfoNCE loss. A well-documented issue is the "modality gap," where embeddings from different modalities occupy distinct, non-overlapping regions of the shared latent space. This paper investigates whether this gap is an inherent byproduct of the data or an active result of the optimization dynamics within the InfoNCE loss function.

The authors hypothesize that the modality gap is a mode-failure caused by the InfoNCE objective when using independent encoders. To test this, they conduct a controlled uni-modal experiment using two independent encoders with identical initializations. By varying the temperature parameter (τ) of the InfoNCE loss, they observe how the embeddings diverge. They then propose xNCE, a modified loss function that samples both inter-modality and intra-modality negative pairs. By including intra-modality negatives, the loss function no longer treats the two modalities as distinct entities to be separated, effectively removing the degenerate divergence objective.

The experiments confirm that the modality gap is an active optimization outcome of InfoNCE at low temperatures. In the uni-modal MNIST experiment, the gap emerges immediately after training begins, even with identical initialization. The proposed xNCE method successfully reduces this gap across all tested temperatures. Furthermore, in multi-modal experiments on the COCO dataset, xNCE maintains competitive retrieval performance while significantly improving zero-shot classification accuracy compared to the standard InfoNCE baseline and other regularization-based approaches.

This work provides a mechanistic explanation for the modality gap, shifting the focus from architectural or data-related causes to the loss function itself. By demonstrating that a simple change to the negative sampling strategy (xNCE) can mitigate the gap while simultaneously improving downstream transfer performance, the paper offers a practical and theoretically grounded path for training more unified multi-modal representations.