Progressive Pixel-Neighborhood Deformable Cross-Attention for Multispectral Object Detection

Multispectral object detection aims to combine visible (RGB) and thermal infrared (TIR) images to maintain high detection performance under challenging lighting conditions. However, existing Transformer-based fusion methods often rely on global cross-attention, which is computationally expensive and prone to noise. Furthermore, these models often struggle with 'weak misalignment'—small, non-linear spatial offsets between sensors that lead to ghosting and localization errors. This paper asks how to achieve high-precision cross-modal alignment while simultaneously improving computational efficiency.

The authors propose PNAFusion, a framework that integrates two key components: the Pixel-Neighborhood Cross-Attention (PNCA) module and the Adaptive Deformable Alignment (ADA) module. Instead of performing global matching, PNCA restricts cross-modal interaction to a local $k \times k$ neighborhood, reducing complexity from quadratic to linear relative to the number of pixels. The ADA module complements this by learning pixel-wise offsets, allowing the model to dynamically warp thermal features to match visible features, effectively correcting non-linear spatial displacements. These modules are integrated into an iterative feedback loop that progressively refines the alignment between modalities.

Experiments on the FLIR, M3FD, and DroneVehicle datasets demonstrate that PNAFusion achieves state-of-the-art accuracy, reaching 90.5 mAP@0.5 on M3FD when using a YOLOv5 detector. The framework provides a superior trade-off between accuracy and resource usage; it reduces allocated GPU memory by 33% and lowers theoretical FLOPs from 194.8 G to 156.4 G compared to the ICAFusion baseline. The authors emphasize that while the method is more efficient in terms of memory and operations, the specific operations of deformable sampling and iterative refinement do introduce additional latency, which should be considered for real-time applications.