Peter R.D. van der Wal, Nicola Strisciuglio, George Azzopardi
4 min
Vision Transformers (ViTs) have become a dominant architecture in computer vision, yet they often struggle with "shortcut learning," where the model relies on spurious background correlations rather than the actual object of interest. The authors investigate whether this issue stems from the standard softmax-based self-attention mechanism, which forces all attention scores to be positive and effectively ignores negative correlations that could help distinguish relevant features from background noise.
Inspired by biological "push-pull" inhibition—where neurons suppress responses to non-preferred stimuli to sharpen contrast—the authors propose Inhibited Self-Attention (ISA). ISA replaces the standard softmax normalization with a competitive mechanism that incorporates both softmax (positive) and softmin (negative) distributions. By subtracting the negative attention scores from the positive ones, the model explicitly learns to inhibit irrelevant background regions, resulting in a more focused and discriminative attention map.
Experiments across ImageNet-1k, COCO, and various robustness benchmarks demonstrate that ISA-equipped ViTs consistently outperform standard ViTs. Key findings include:
This work addresses a fundamental limitation in how Transformers process visual data. By proving that negative attention scores contain valuable information for feature selection, the authors provide a simple, biologically-inspired path toward more interpretable and robust vision models. This approach is particularly valuable for real-world applications where models must distinguish between subtle object features and distracting background noise without the need for complex auxiliary architectures.
Vision Transformers (ViTs) have demonstrated remarkable performance in computer vision tasks. However, their self-attention mechanism often diffuses focus across background regions, relying on spurious correlations rather than object-relevant cues. Inspired by inhibitory mechanisms observed in biological vision systems, we propose the Inhibited Self-Attention (ISA), a novel self-attention that integrates inhibitory signals to enhance feature selectivity and suppress spurious responses. In contrast to conventional self-attention, which relies solely on positive attention values due to softmax normalization, our approach retains and utilizes negative attention scores to suppress irrelevant features and sharpen focus on objects of interest. Experiments across multiple datasets, including ImageNet-1k and COCO, and several robustness benchmarks demonstrate that ISA enhances object-centric selectivity, reduces shortcut reliance, and improves out-of-distribution generalization. Our analysis of relevance maps confirms that ViTs with ISA exhibit sharper, more localized focus on object-relevant regions while reducing distractions from non-relevant (background) features, enabling more reliable models. We release our code at https://github.com/prdvanderwal/inhibited-self-attention
Alex: Right. But here's the key insight from this paper. When softmax runs, it also produces negative scores for parts of the image that seem unimportant. Standard models simply discard those negative scores. This new system keeps them and runs them through a second function — called "softmin" — which does the reverse of softmax. Instead of highlighting the highest scores, it highlights the lowest ones. In other words, it maps out the background.
Sam: So softmax says "look here," and softmin says "definitely not there" — and the model subtracts one from the other?
Alex: That's exactly it. The result is that the background gets suppressed and the object of interest gets a cleaner, stronger signal. And crucially, this doesn't require any additional computing power or extra parameters. It's purely a change in how the model processes information it was already receiving.
Sam: So they're not building a bigger engine — they're just using the existing one more efficiently.
Alex: That's a good way to frame it. The information about what to ignore was always there in those negative scores. It was just being thrown away. This approach recycles it.
Sam: Did the researchers have a way to actually measure whether the model is looking in the right place?
Alex: They did. They introduced a metric they call "Attention-on-Objects." It works by checking where the model's focus lands within an image and measuring how much of that focus falls on the actual object versus the surrounding background. They found that models using Inhibited Self-Attention consistently directed their attention toward the object, while standard models remained much more scattered.
Sam: So it's not just a theoretical improvement — they could actually observe the difference in where the model was looking.
Alex: Correct. And that's what makes the metric useful. It gives you a direct window into the model's behaviour, rather than just measuring whether it got the final answer right.
Sam: It seems like a meaningful step toward AI that's more reliable when it encounters unfamiliar images. Thanks for walking through the logic, Alex.
Alex: It's a clear example of how rethinking an existing process — rather than simply adding more complexity — can lead to a more disciplined system. Thanks for listening to ResearchPod.