Align and Segment: Unsupervised Learning for Building Segmentation From Misaligned Labels

In remote sensing, training deep learning models for semantic segmentation is often hindered by spatial misalignments between imagery and available label sources (e.g., OpenStreetMap). Standard supervised learning fails in these scenarios because models tend to learn the misaligned labels directly, leading to poor performance. This paper asks: can we jointly learn to align misaligned labels and perform accurate semantic segmentation without any golden (perfectly aligned) ground-truth labels?

The authors propose the Align and Segment (AnS) framework, which consists of two primary components: a Segmentation Network (SNet) and a Transformation Network (TNet). The TNet uses a spatial transformer module to predict an affine transformation that corrects the misalignment of the labels, while the SNet learns to segment the image. To prevent the model from falling into a trivial solution—where the network simply learns to predict the misaligned labels—the authors introduce a self-supervised consistency loss. This loss forces the TNet to predict consistent transformations even when random perturbations are applied. Furthermore, they employ geometric data augmentations (like rotations and flips) to neutralize systematic biases in the training data.

The AnS approach successfully disentangles the alignment task from the segmentation task. Experimental results on both synthetic datasets (with controlled noise) and real-world datasets (e.g., OpenStreetMap data for San Juan and the ReBO dataset) demonstrate that the model achieves high-quality segmentation and precise alignment simultaneously. The method outperforms existing unsupervised and semi-supervised baselines, proving that it can effectively utilize large-scale, noisy, and misaligned datasets that were previously underutilized due to curation costs.

This research significantly lowers the barrier for using crowd-sourced or legacy geospatial data for training high-performance computer vision models. By eliminating the need for expensive, manually corrected golden labels, the AnS framework allows researchers to leverage massive, existing datasets for urban planning, disaster response, and environmental monitoring in regions where high-quality annotated data is scarce.