CouCE: A Unified Causal Framework for Debiased Deep Metric Learning

Deep Metric Learning (DML) models often suffer from shortcut learning, where they capture spurious correlations (e.g., background context) or nuisance variations (e.g., pose, lighting) rather than true semantic similarity. The authors investigate how to unify the treatment of these two distinct causal confounders—background spurious correlations and foreground nuisance perturbations—within a single, efficient DML architecture.

The authors propose Counterfactual Causal Embedding (CouCE), a framework built on a Structural Causal Model (SCM) that identifies two interference pathways: a backdoor path from background context and a direct nuisance path from foreground variations. CouCE introduces two primary modules:

• Orthogonal Dictionary-Based Backdoor Adjustment (ODBA): This module maintains a variance-gated dictionary of background features. It uses soft orthogonal regularization to force the model to learn embeddings that are independent of the background subspace, effectively blocking the backdoor path without needing spatial annotations.
• Multi-Scale Randomized Causal Intervention (MSRCI): This module targets foreground nuisances by applying multi-scale Fourier amplitude randomization to feature maps. By perturbing amplitude (which encodes style/illumination) while preserving phase (which encodes structure), the model is forced to maintain consistent proxy similarity rankings, enforcing causal invariance.

These modules are integrated into a unified training objective that includes a covariance penalty to ensure dimensional independence. Crucially, both modules are discarded at inference time, resulting in zero architectural overhead.

CouCE consistently outperforms state-of-the-art DML methods across three major benchmarks: CUB-200-2011, Cars-196, and Stanford Online Products. The framework demonstrates robust zero-shot generalization by successfully disentangling semantic features from both background and foreground confounders. Ablation studies confirm that the two modules are complementary, with ODBA providing significant gains in background-heavy datasets like CUB, and MSRCI excelling in datasets with high foreground variance like Cars-196. The authors also show that CouCE is plug-and-play, providing consistent performance improvements when integrated with various existing proxy-based loss functions.