LISA: LIKELIHOOD SCORE ALIGNMENT FOR VISUAL-CONDITION CONTROLLABLE GENERATION

Visual-condition controllable generation often uses a dual-branch architecture: a frozen main network (providing unconditional priors) and a trainable side network (encoding conditions). While effective, the side network's role is often poorly defined, leading to inefficient training. This paper introduces Likelihood Score Alignment (LISA), a regularization technique that explicitly forces the side network to learn a likelihood score—the mathematical bridge between unconditional and conditional generation.

LISA treats the conditional generation process as a decomposition of scores. By applying Bayes' rule, the authors identify that the side network should ideally represent the likelihood score (the difference between the conditional and unconditional scores). To enforce this, LISA uses a lightweight decoder to project intermediate features from the side network into a latent score space. This projection is then regularized against an approximated likelihood score, calculated as the difference between the known conditional score (from training data) and the unconditional score (from the frozen main network).

Experiments across various tasks—including pose, depth, and segmentation-conditioned generation—demonstrate that LISA significantly accelerates convergence. For instance, in depth-conditioned tasks, models using LISA achieved superior metrics in 4,000 iterations compared to standard baselines requiring 10,000. Because the auxiliary decoder is only used during training and discarded at inference, LISA provides these performance gains with zero additional inference cost.