Shuang Liang, Lejun Liao, Shiyuan Zhang, Max C. Zhang, Xiaolong Luo, Han Wang, Stefano Anzellotti, Yuan Yuan
9 min
Abstract
Given a target dataset, such as faces with eyeglasses, and a background dataset, such as faces without, contrastive analysis separates \textit{salient} factors specific to the target from \textit{common} content shared by both. We aim for salient representations that capture target-specific detail in each image, such as the shape, color, and position of the glasses, so that they reveal subtypes without subtype labels and guide the generation of new examples of a discovered subtype, even one with no name or text description. We introduce SAGE, which learns both factors directly in the high-dimensional spatial latent of a frozen representation autoencoder and conditions a diffusion transformer on the learned salient representation of a reference image. On Digits-ImageNet and FFHQ eyeglasses, SAGE combines high-fidelity \textit{reconstruction} (rFID below $2$) with unsupervised \textit{subtype discovery}, recovering the digits better than baselines (probe accuracy $0.950$ vs.\ at most $0.281$) and revealing eyewear types, finer sunglasses styles, and mislabeled images; salient-conditioned \textit{generation} raises Digits-ImageNet subtype accuracy over the unfactorized latent ($90.5\%$ vs.\ $27.7\%$) and diversity on both datasets. On retinal OCT, SAGE's salient space separates three diseases using only normal/disease labels.
Sam: That gives the split a test beyond reconstruction. But how do they judge a swap without a matching photograph of the edited face?
Alex: They train discriminators, networks that distinguish swapped outputs from reconstructed images in each group. Removing glasses should produce a background-like face; adding glasses should produce a target-like face. Using reconstructions for comparison helps prevent decoder artifacts from becoming the discriminators' shortcut.
Sam: That could still produce generic glasses. What makes the transferred pair belong to the particular reference image?
Alex: They re-encode swapped images and require recovery of the factors that made them. The salient factor should survive its move onto a different face. They also penalize unnecessary salient content, encouraging a sparse representation. Additional consistency and contrastive losses discourage shared content from appearing in both parts.
Sam: So the model has to carry enough detail to recognize the transferred attribute, but avoid carrying the donor's whole face.
Alex: In the authors' examples, thick black frames move onto a different face while its identity remains. A transferred digit keeps its stroke shape and location. Removing the salient factor erases the glasses or digit while retaining the surrounding content. These are qualitative demonstrations, backed by quantitative tests.
Sam: Let's take the strongest test. How well does the representation recover categories it never received during training?
Alex: On photographs with added handwritten digits, evaluation uses roughly twenty-five thousand target images. A linear probe—a simple classifier trained on frozen representations—recognizes digit identity with ninety-five percent accuracy from SAGE's salient part. The strongest baseline reaches about twenty-eight percent.
Sam: That probe uses digit labels, though. We should separate learning without labels from evaluating with them.
Alex: Yes. Subtype labels enter evaluation, not SAGE's representation training. The authors also measure clustering agreement with digit labels, and SAGE improves that comparison substantially. But the digit groups partly overlap. Strong probe accuracy doesn't mean the model found perfectly isolated clusters.
Sam: Does it actually preserve reconstruction, or does cleaner discovery come at the expense of the image?
Alex: Reconstruction Fréchet Inception Distance, a distribution-based measure where lower is better, stays below two on both natural-image datasets. The generative baselines trained from scratch exceed one hundred twenty. Much of that advantage comes from reusing the pretrained decoder, rather than learning image reconstruction from scratch.
Sam: So that comparison demonstrates the value of the whole system, not just the separation objective.
Alex: The authors also test SepCLR on the same frozen backbone and encoder architecture. SAGE still does much better on digit discovery. However, the evaluation pooling and representation dimensions differ. And removing the swap objective makes digit discovery largely collapse, supporting swapping as a necessary ingredient here.
Sam: Digits are deliberately added. Did the face experiments uncover anything less predetermined?
Alex: They recovered reading glasses and sunglasses, plus finer sunglasses groups described as wayfarer-like, aviator-like, and mixed styles. Another group contained faces incorrectly annotated as wearing glasses. That prompted a manual audit of roughly thirty-four hundred target validation images, conducted without showing annotators the model's assignments.
Sam: Did the model identify all annotation mistakes, or did the audit go further?
Alex: The audit went further. SAGE surfaced most of the no-glasses cases as a separate group. Corrections between reading glasses and sunglasses came from the human audit, not SAGE. Some face paint and eye masks still fell among genuine glasses, showing where appearance-based grouping can mislead.
Sam: Now generation. If the model uses a reference image, how do we know it isn't simply copying that image?
Alex: A diffusion transformer—a generator that transforms noise into image representations—receives the pooled salient factor. Pooling removes location while retaining active feature channels. The comparison generator instead receives the full, unfactorized representation. Both use the same references and initial noise samples.
Sam: Then the practical test is whether the digit stays while the photograph changes.
Alex: On the digit dataset, salient conditioning preserves the reference digit in about ninety percent of generated images, versus about twenty-eight percent with the full representation. It also produces substantially more variation within each reference condition. On faces, both preserve eyewear type well, but salient conditioning produces more varied faces.
Sam: Where does the claim stop? Does the model completely remove target information from the common part?
Alex: No. The authors explicitly distinguish a clean salient factor from complete separation. Digit identity remains decodable from the common representation, even when common-only decoding removes the visible digit. Group labels don't uniquely determine how correlated attributes should be divided. Also, quantitative clustering uses the known number of evaluation categories.
Sam: And the retinal results—do they establish something clinically useful?
Alex: They show disease-category structure learned from normal-versus-disease labels, evaluated on patients absent from training. The edits suggest disease-related features, but the authors say they are not clinical evidence. Reconstruction is less faithful there, because the frozen encoder and decoder inherit natural-image training priors.
Sam: Who should read the full document, and where should they start?
Alex: Researchers in contrastive analysis, image representations, or reference-conditioned generation should read Salient Factor Discovery and the generation comparison first. Then read Limitations to distinguish useful separation from complete disentanglement. Biomedical readers should also inspect the retinal preprocessing, which addresses non-anatomical white borders. For everyone else: learning the attribute separately can let you generate its examples without first giving it a name.
Sam: Keep that distinction in mind when you read.