The Impact of VAE Design on Latent Pose Representations for Diffusion-based Sign Language Production

Latent diffusion models have become a standard approach for Sign Language Production (SLP), where a two-stage pipeline is typically employed: first, an autoencoder compresses skeletal pose sequences into a latent space, and second, a diffusion model learns to generate these latent representations conditioned on text. While researchers often optimize the autoencoder solely for reconstruction accuracy (how well it recovers the original pose), it remains unclear if this metric is the best indicator of how well the latent space supports subsequent generative modeling.

The authors investigate how architectural and training choices in a VAE affect the structure of the latent space and, consequently, the performance of a text-to-sign latent diffusion model. They evaluate four VAE variants:

• BaseVAE: A simple MLP-based architecture with a standard position-based reconstruction loss.
• StructVAE: Incorporates Graph Convolutional Networks (GCNs) and temporal blocks to better capture skeletal and motion dynamics.
• MultiObjVAE: Adds multi-objective losses, including velocity penalties and region-specific weighting, to emphasize fine-grained movements like mouth and finger articulation.
• FactorVAE: Extends the multi-objective approach by predicting separate latent distributions for different body regions (torso/arms, hands, face).

They analyze these models using both standard SLP metrics (like Mean Joint Error and back-translation BLEU scores) and novel latent space metrics, such as effective dimensionality and inter-dimensional correlation.

The experiments reveal that reconstruction accuracy is an incomplete proxy for generative success. While more complex architectures (like FactorVAE) improve the quality of the latent space, the relationship between latent structure and diffusion performance is nuanced. Specifically, high latent space redundancy (measured by correlation) can sometimes facilitate optimization, while effective dimensionality provides a better indicator of whether the latent space contains the rich information necessary for high-quality sign generation. The authors demonstrate that variations in back-translation BLEU scores—a key measure of semantic sign quality—are more closely tied to these latent structural properties than to simple reconstruction error.