VARIATIONALLOSSYAUTOENCODER

Research Question and Approach

Representation learning aims to disentangle the underlying causal factors of data, such as separating global object identity from local texture. Standard Variational Autoencoders (VAEs) often struggle with this because powerful decoders (like RNNs or PixelCNNs) tend to ignore the latent code entirely, modeling the data distribution directly. This paper introduces the Variational Lossy Autoencoder (VLAE), which addresses this by using an information-theoretic perspective on VAEs to explicitly control what information is stored in the latent representation.

The Information Preference Property

The authors identify an 'information preference' phenomenon: when a decoder is powerful enough to model the data locally, it will do so, rendering the latent code redundant. To counteract this, the VLAE uses a decoder with a restricted receptive field (a small-window PixelCNN). Because this decoder cannot model long-range dependencies, it is forced to ignore global structure, which the model then captures in the latent code. This creates a 'lossy' autoencoder that intentionally discards local, high-frequency information like texture while preserving global structure.

Improving Generative Performance

Beyond representation learning, the authors improve density estimation by replacing the standard Gaussian prior with an autoregressive flow. This approach, which they term an Autoregressive Flow (AF) prior, increases the expressiveness of the generative model. They demonstrate that this is mathematically equivalent to using an Inverse Autoregressive Flow (IAF) posterior but offers a more efficient generative path, leading to state-of-the-art results on MNIST, OMNIGLOT, and Caltech-101 Silhouettes.

This work provides a principled way to design VAEs that perform both high-quality density estimation and meaningful representation learning. By manipulating the decoder's architecture, researchers can dictate the level of abstraction in the learned latent space, making it a powerful tool for tasks where global structure is more important than pixel-perfect reconstruction.