Convolutional Neural Shading for High-Quality 3D Reconstruction from Multi-View Images

Traditional 3D reconstruction methods often rely on single-point geometric information, such as surface positions and normals, to predict color and shape. This approach frequently fails to capture local geometric details, leading to artifacts in textureless or dark regions and discontinuities at image boundaries. The authors seek to overcome these limitations by incorporating spatial context into the neural rendering pipeline.

The proposed Convolutional Neural Shading (CNS) pipeline replaces standard MLP-based rendering with a two-part neural architecture. First, a fine-detail displacement network learns to refine vertex positions and surface normals by correlating neighboring values in rendering coordinates. Second, a convolutional neural shader predicts pixel-wise colors by leveraging this refined local geometry and spatial context, rather than treating each point in isolation. The system jointly optimizes the mesh geometry and the neural shading network using a combination of mesh-based losses (Laplacian and normal consistency) and image-based losses (RGB, mask, SSIM, and LPIPS).

Experimental results across the DTU, BlendedMVS, and THuman datasets demonstrate that CNS significantly outperforms state-of-the-art methods like Neuralangelo. By explicitly modeling local spatial correlations, the model produces higher-fidelity surfaces, particularly in challenging areas such as dark or textureless regions. Furthermore, the fine-detail displacement network effectively mitigates surface irregularities and discontinuities that typically occur at image boundaries, resulting in more accurate and visually consistent 3D reconstructions.

This work addresses a fundamental bottleneck in neural differentiable rendering: the inability of point-wise MLP models to account for local neighborhood information. By demonstrating that convolutional architectures can be integrated into the rendering pipeline to capture spatial context, this research provides a viable path toward higher-quality 3D content generation for industries like film and gaming, where manual 3D modeling is labor-intensive and costly.