Efficient Remote Sensing Instance Segmentation with Linear-Time State Space Distilled Visual Foundation Models

Remote sensing instance segmentation requires processing high-resolution imagery, which often leads to massive computational costs when using standard Transformer-based architectures. The quadratic complexity of self-attention mechanisms makes these models impractical for resource-constrained environments, such as edge devices or in-orbit satellite processing. This paper investigates whether knowledge distillation can effectively transfer the rich feature representations of large Vision Transformer (ViT) foundation models into compact, linear-time State Space Models (SSMs) without sacrificing segmentation performance.

The authors propose RS4D, a two-stage training framework. First, they perform knowledge distillation by training a lightweight SSM-based student encoder to mimic the output of a pre-trained ViT-based teacher (the Segment Anything Model, or SAM). This stage uses an adaptive noise and masking strategy to compress the teacher's vast self-attention knowledge into a dense, linear state space. Second, the distilled encoder is fine-tuned on specific remote sensing datasets. The authors explore three distinct SSM backbone architectures—VanillaMamba, TransMamba, and ScanningMamba—to identify the most effective configurations for dense prediction tasks in remote sensing.

RS4D demonstrates that linear-time SSMs can serve as highly efficient alternatives to Transformers for dense visual tasks. Experimental results on the SSDD, WHU, and NWPU datasets show that the proposed SSM backbones achieve significant efficiency gains: an 8x reduction in parameters and a 9x reduction in FLOPs compared to ViT-based methods. Despite this drastic reduction in computational overhead, the model maintains or exceeds the accuracy of both traditional CNN-based and modern ViT-based instance segmentation approaches, providing a more viable solution for time-sensitive remote sensing applications.