Leveraging Dissimilarity Invariance as a Robust Anchor for Learning with Noisy Labels

Deep learning models often suffer from performance degradation when trained on datasets with noisy labels. This is largely because noisy labels distort the learned semantic similarities between samples, causing the model to internalize false associations. This paper investigates whether there exists a more robust structural property in data that remains stable under label corruption, and how such a property can be used to improve model training.

The authors introduce the concept of Dissimilarity Invariance, observing that while similarities between semantically related samples are fragile under label noise, the dissimilarity between semantically unrelated samples (negative pairs) remains remarkably stable. Based on this, they propose NegScale, a plug-and-play framework that shifts the training focus from fragile similarity to robust dissimilarity. NegScale integrates two main components:

• Structured Negative Orthogonality Penalty (SNOP): Enforces subspace orthogonality for unrelated samples to ensure they remain distinct, using both global and confidence-weighted local penalties.
• Dissimilarity-Calibrated Similarity Adjustment (DCSA): Uses dissimilarity anchors to identify and suppress spurious similarities, preventing the model from overfitting to noisy positive pairs.

The authors provide theoretical analysis to justify why negative-pair similarity is more invariant to gradient updates and how NegScale bounds feature perturbation under noisy conditions.

Empirical evaluations on CIFAR-10, CIFAR-100, and real-world noisy datasets (CIFAR-N, Animal-10N) demonstrate that NegScale consistently outperforms state-of-the-art baselines. The results confirm that by explicitly modeling dissimilarity and suppressing noisy similarity, the framework maintains higher accuracy across various noise types and rates. The theoretical analysis supports these findings, showing that the proposed regularization effectively reduces the impact of label noise on feature representations.