The Trade-off Between Covariate Dependence and Latent Structure in Representation Learning

Disentangled representation learning aims to map high-dimensional data to latent spaces where individual dimensions align with distinct, semantically meaningful covariates. While unsupervised methods target latent dimension independence without guaranteeing semantic alignment, supervised approaches use observed covariates to structure the latent space. However, when covariates are correlated—a common scenario in real-world data—these methods struggle to simultaneously achieve one-to-one latent-covariate alignment and latent independence.

This paper investigates the fundamental tension between latent structural constraints and latent-covariate dependence. The authors ask how one can systematically control representations when covariates are correlated, closing a gap in existing linear and nonlinear representation learning frameworks.

Leveraging the inherent non-identifiability of latent representations, the authors formulate optimization problems over invertible transformations of the latent space. They define and analyze four distinct disentanglement regimes:

1. Independent latent dimensions, which whitens the latent space while maximizing alignment.
2. Strong latent dependence on covariates, which prioritizes alignment at the cost of relaxing latent independence.
3. Intermediate latent inter-dependence, which interpolates continuously between independence and unconstrained dependence using a weighting parameter.
4. Exclusive latent dependence, which forces each dimension to correlate solely with its own designated covariate.

Through rigorous mathematical analysis, the authors prove that these regimes are strictly ordered by their latent-covariate alignment strength. They show that enforcing strict independence or exclusive dependence incurs a provable cost in alignment capability, making the trade-off mathematically precise.

To translate these theoretical insights into practical tools, the authors introduce two main approaches. First, they apply their derived transformations post-hoc to realign the latent representations of pretrained models such as CLIP, DINOv2, and Vision Transformers (ViT). Second, they integrate these transformations into informed Factor Analysis (iFA), a novel probabilistic graphical model that couples latent factors directly with observed covariates under automatic relevance determination priors.

Experiments on simulated covariate scenarios and real multi-omics data demonstrate that both post-hoc alignment and iFA successfully restore controllability and interpretability to structured latent representations under correlated covariates.