BEYOND OBJECTIVE EXPRESSIVITY: GEOMETRY PRESERVATION IN MULTIMODAL CONTRASTIVE LEARNING

Multimodal contrastive learning often involves aligning three or more modalities, but researchers have largely focused on designing more expressive objective functions to handle this complexity. This paper investigates a complementary, often overlooked factor: the geometric properties of the modality-specific encoders themselves. The authors hypothesize that the trainability and ultimate performance of these models are fundamentally constrained by the conditioning of the encoder's Jacobian, which dictates how information and gradients propagate during training.

The authors analyze the Jacobian condition number—the ratio of the maximum to minimum singular values—of modality encoders. They demonstrate that standard ReLU-based MLP encoders often suffer from "geometric degeneration," where the minimum singular value collapses and the maximum singular value explodes, leading to poor optimization. To address this, they introduce Geometry-Preserving Encoders (GPEs), which incorporate residual transport paths and LeakyReLU activations to maintain well-conditioned Jacobians. They validate this approach across a synthetic benchmark (Synthetic-XNOR) and four real-world datasets, including the UK Biobank and MIMIC-IV, comparing GPEs against standard encoders across various contrastive objectives.

The study finds that standard encoders frequently exhibit exploding Jacobian condition numbers, which directly correlates with degraded retrieval and linear probe performance. By contrast, GPEs stabilize the training dynamics by preventing singular-value collapse and explosion. Crucially, the authors show that while more expressive contrastive objectives may improve retrieval performance, they do not necessarily improve downstream linear probe performance if the underlying encoders are poorly conditioned. GPEs, however, consistently boost performance across both metrics, suggesting that encoder architecture is as critical as objective design for successful multimodal alignment.