Consistent Evidence, Robust Recognition: Faithful Attribution Regularization under Geometric Transformations

Deep learning models often exhibit inconsistent evidence reliance when subjected to label-preserving geometric transformations (e.g., rotations or flips). While attribution methods can visualize this, they are typically used for post-hoc diagnosis rather than as a training signal. This paper asks: how can we leverage faithful, decision-linked attribution to regularize a model, ensuring it relies on spatially consistent evidence across transformations without sacrificing predictive performance?

The authors introduce an annotation-free framework that treats the attribution process as a target for regularization. Instead of using standard gradient-based saliency maps (like Grad-CAM), which often lack faithfulness, the authors employ a submodular search algorithm. This algorithm greedily selects compact, class-discriminative image regions that contribute most to the model's prediction.

To make this discrete, path-dependent search process differentiable, the authors introduce a Submodular Ranking Loss. This loss consists of two components:

1. Path-Consistency: It forces the model to rank spatially corresponding regions similarly across both original and transformed search trajectories.
2. Termination-Alignment: It ensures that the transformed search process reaches the same stopping criterion as the original, effectively regularizing the entire evidence-selection trajectory.

Experiments on ImageNet-100 and ImageNet-1K demonstrate that the proposed method significantly improves attribution stability and faithfulness (e.g., improving Insertion and Deletion metrics) across various architectures like ViT-B/16, ResNet-50, and ConvNeXt-B. Crucially, this improvement in evidence consistency is achieved with minimal impact on the model's clean-image classification accuracy (typically a drop of less than 0.30 percentage points). The results suggest that by aligning the underlying decision process rather than just the final heatmap, models become more robust to geometric transformations.