ResearchPod Summary
Synthesizing photorealistic novel views of human faces at high resolutions while preserving identity, fine appearance details, and cross-view geometric consistency remains a core challenge in computer vision. Existing approaches often rely on diffusion models requiring heavy 2D image pre-training, or struggle with data scarcity and realism on non-frontal camera angles. This paper investigates whether next-scale autoregressive transformers can serve as an effective backbone for scalable, multi-output human view synthesis under data-scarce conditions, eliminating the need for massive 2D generative pre-training.
Building upon the next-scale autoregressive paradigm (specifically VAR and ArchonView), the authors extend the architecture to simultaneously handle multiple input views and generate multiple output views in a single forward pass. Instead of raster-scanning tokens sequentially, the model generates entire image scales progressively from low to high detail. The pipeline couples this next-scale transformer with an off-the-shelf pixel-aligned 3D Gaussian lifting model to produce accurate 3D face representations. Because training a high-resolution model from scratch on scarce realistic data is impractical, the authors introduce a three-stage training strategy: first adapting a pre-trained object-level model to 512x512 resolution using a large proprietary object dataset, then fine-tuning on a human head dataset using single or multiple loosely-posed inputs, and finally specializing the network to simultaneously output six canonical views.
Empirical evaluations demonstrate that the proposed next-scale autoregressive framework achieves superior perceptual fidelity, sharpness, and cross-view coherence compared to existing state-of-the-art baselines like FaceLift and VoluMe. By processing scales sequentially across all target views in one forward pass, the model captures strong cross-view correlations. Furthermore, the multi-input variant that accepts several loosely-posed selfie-like images benefits from additional conditioning to yield even more realistic 3D face reconstructions.
This work demonstrates that next-scale autoregression is a powerful, data-efficient alternative to diffusion models for human-centric novel view synthesis and 3D reconstruction. By circumventing the requirement for large-scale 2D pre-training and massive synthetic datasets, the method offers a practical foundation for real-world applications such as virtual reality, gaming, and avatar creation where high-fidelity facial consistency from sparse captures is critical.
AI-generated third-party summary by ResearchPod. Not official content or an endorsement by the paper authors or affiliated organizations.