Norm or Direction? Decoding Vision Mambas for High-Resolution Vision

This paper investigates the representational differences between Vision Mamba (VMamba) and MambaOut, a Gated CNN-based model. While MambaOut has been shown to match or exceed VMamba in standard image classification, VMamba maintains a performance advantage in high-resolution dense prediction tasks. The authors seek to determine whether these models organize visual information differently and how these organizational strategies influence performance across varying task complexities.

To compare the internal representations, the authors employ Centered Kernel Alignment (CKA) to analyze block-level similarity. They further decompose spatial tokens into magnitude and direction to assess where class-discriminative information is stored. By comparing token-level norms with Grad-CAM attribution and performing token-replacement experiments, they evaluate how each model aggregates evidence. Finally, they test these models on high-resolution classification and semantic segmentation to see if the identified representational differences correlate with performance in dense prediction settings.

• Representational Divergence: CKA analysis reveals that VMamba’s final stage blocks form representations distinct from both MambaOut and its own earlier layers, suggesting a unique, task-specific adaptation.
• Encoding Strategies: MambaOut relies on high-norm foreground tokens to store class-discriminative information, which aligns with Grad-CAM heatmaps. In contrast, VMamba’s high-norm tokens are often found in background regions; instead, it encodes discriminative signals primarily through the directional (angular) component of its tokens.
• VMamba’s strategy of distributing logit support broadly across spatial tokens makes it more stable than MambaOut as token counts increase. This explains why VMamba consistently outperforms MambaOut in full fine-tuning for semantic segmentation, where dense, spatially selective representations are required.