Encoder–Decoder Manifold Alignment for Idempotent Generation

Modern generative models often rely on encoder-decoder architectures to map data into a latent space for manipulation. A persistent problem in these models is that repeated applications of the encoder-decoder pipeline—such as during iterative editing—often lead to "drift," where the output progressively degrades or loses its original identity. The authors investigate whether this failure stems from a geometric mismatch between the encoder's latent manifold and the decoder's reconstruction manifold, and whether enforcing idempotency can resolve this.

To address the geometric mismatch, the authors introduce a regularization term that forces the encoder and decoder to learn consistent representations of the same data manifold. By requiring that the encoder produces the same latent representation before and after a decoding step (i.e., E(D(E(x))) ≈ E(x)), the model is encouraged to behave as a stable projection operator. A critical component of this framework is the use of a stop-gradient operator on the initial latent code, which prevents the model from collapsing into a trivial identity mapping while ensuring the encoder and decoder remain mutually compatible.

Empirical results demonstrate that the proposed alignment framework significantly reduces idempotency error across various architectures, including VAEs, VQ-VAEs, and latent diffusion models. In image generation tasks, the model maintains high visual fidelity and identity preservation even after dozens of repeated applications, whereas baseline models exhibit significant structural drift and artifact accumulation. Furthermore, in image editing benchmarks, the idempotent models show superior performance in following instructions while better preserving the source image's underlying structure and fine-grained details.

This work provides a principled, model-agnostic way to improve the stability of generative models. By treating idempotency as a necessary condition for optimal projection, the authors offer a solution that enhances the reliability of latent-space editing, making generative models more controllable and robust for practical applications where iterative refinement is required.