Contrastive-Collapsed Loss for Flexible and Geometrically Optimal Embeddings and Faster Convergence

Neural networks often exhibit a phenomenon known as Neural Collapse (NC) during the final stages of training, where class representations collapse into a fixed geometric structure called a Simplex Equiangular Tight Frame (ETF). While some existing methods attempt to force this structure by aligning features to fixed, predefined targets, these approaches can be rigid and limit the flexibility of the learned embeddings. The authors introduce CoCo (Collapsing and Contrasting) loss, a new objective function designed to learn these well-structured representations in a data-adaptive manner. Instead of imposing static targets, CoCo uses pairwise constraints to enforce unit-norm normalization, intra-class attraction, and inter-class separation, allowing the network to discover the optimal geometric configuration for the specific dataset.

The authors demonstrate that minimizing CoCo loss naturally induces the properties of Neural Collapse, including the formation of a simplex ETF. Unlike methods that rely on fixed kernels (like kernel SVMs) or fixed target vectors (like Dot Regression), CoCo acts as a flexible, data-driven kernel that adapts to the intrinsic structure of the data. Theoretical analysis suggests that CoCo benefits from more informative gradients and a closer initialization to the optimal geometric configuration. Empirical results on diverse tabular datasets from the OpenML-CC18 benchmark show that CoCo achieves competitive predictive performance while promoting tighter class clustering and faster convergence compared to standard cross-entropy and other geometric baselines.

CoCo provides a bridge between probabilistic classification and geometric representation learning. By treating the embedding function as a learnable kernel, it offers a more flexible alternative to traditional kernel methods that require manual selection of kernel functions. Its ability to achieve faster convergence and robust class separation makes it a promising objective for tasks where both discriminative power and geometric interpretability are required. Furthermore, the authors provide a balanced variant of the loss to handle class-imbalanced datasets, increasing its practical utility in real-world classification scenarios.