Reference-Based Face Super-Resolution Using the Spatial Transformer

Face super-resolution is an ill-posed problem where a low-resolution input can map to many possible high-resolution outputs. This paper addresses the challenge of using high-resolution reference images of the same person to guide the reconstruction process. The authors specifically aim to improve the stability of the image alignment process—a common bottleneck in existing reference-based models—and to create a more efficient, lightweight architecture.

The authors propose the Face Super-Resolution using the Spatial Transformer (FSRST) model. The core innovation is the Spatial Transformer Alignment (STA) module, which replaces the commonly used deformable convolutions. Deformable convolutions are known to be difficult to train and prone to instability; the spatial transformer, by contrast, uses a localization network to predict affine transformation parameters, providing a more stable and differentiable way to align reference features with the input.

Additionally, the model includes a Distance-Based Weighted Aggregation (DWA) module. This module calculates the L2-distance between the input and reference features to dynamically weight the contribution of each reference image. If no suitable reference is found, the module effectively suppresses the reference input, allowing the model to fall back on single-image super-resolution without performance degradation.

FSRST demonstrates that using a spatial transformer for alignment is not only more stable during training but also highly effective for super-resolution tasks. The model achieves state-of-the-art results on multiple datasets, including DeepFakeDetection and CelebAMask-HQ, while maintaining a significantly smaller parameter count compared to previous methods like MRefSR or C2-Matching. The authors show that their aggregation strategy successfully handles varying degrees of similarity between reference images and the input, providing a robust framework for real-world applications like video conferencing.