β-VAEs as Effective Theories: Tolerance-Dependent Dimension

Research Question and Approach

This paper investigates whether the 'spectral cutoff' logic—where regularization strength ($eta$) in a VAE acts as a filter that collapses latent coordinates based on their reconstruction utility—holds true for nonlinear, fully connected VAEs. In linear Gaussian VAEs, this relationship is exact and aligns with Principal Component Analysis (PCA). The author tests this by training VAEs on the WorldClim dataset, varying the regularization strength, and measuring how latent coordinates collapse and contribute to reconstruction error.

Findings: Nonlinear Spectral Deformation

The study finds that while nonlinear interactions shift and broaden the onset of coordinate collapse, the fundamental ordering of reconstruction utility is preserved. The author introduces a scale-invariant signal fraction as an order parameter to rank latent coordinates. By measuring the marginal utility of these coordinates through ranked pruning, the paper demonstrates that the spectral cutoff remains a valid tool for identifying the most important latent variables.

The Head-Tail Tradeoff

A key contribution is the analysis of 'effective dimension'—the number of latent variables required to reach a specific reconstruction tolerance. The results show a clear head-tail tradeoff: increasing the depth of the VAE architecture concentrates more reconstruction utility into the leading (head) latent coordinates, making the model more efficient at low dimensions. However, this often leads to a higher residual distortion floor in the tail, suggesting that deeper models do not uniformly improve reconstruction across all scales.

This work provides a principled framework for understanding latent space sparsity. Instead of treating the bottleneck size as a fixed hyperparameter, researchers can use the spectral cutoff to determine the effective dimension based on their specific reconstruction tolerance. This approach shifts the focus from finding a single 'optimal' latent dimension to understanding how different architectures redistribute information across the latent spectrum.