Nian Liu, Ni Zhang, Kaiyuan Wan, Ling Shao, Junwei Han
8 min
Salient object detection (SOD) has long been dominated by convolutional neural network (CNN) architectures. While these methods achieve high performance, they are inherently limited by their local receptive fields, which make it difficult to capture the global long-range dependencies often required for identifying salient objects. The authors propose the Visual Saliency Transformer (VST) to shift from this CNN-centric paradigm to a sequence-to-sequence perspective, utilizing a pure transformer to model global context directly.
VST treats images as sequences of patches, processing them with a transformer encoder inspired by T2T-ViT (Tokens-to-Token Vision Transformer). To bridge the gap between low-resolution transformer tokens and the high-resolution requirements of dense prediction tasks, the authors introduce several key innovations:
For RGB-D SOD, the model adds a Cross Modality Transformer (CMT) to integrate depth and color information via cross-attention layers, ensuring that depth cues contribute effectively to the final prediction.
By outperforming current state-of-the-art CNN-based methods across numerous benchmark datasets, VST demonstrates that pure transformer architectures can effectively handle dense prediction tasks. This model provides a new paradigm for vision transformers, showing that sophisticated decoder designs and token-upsampling strategies can overcome the inherent limitations of coarse token sequences in dense computer vision applications.
Existing state-of-the-art saliency detection methods heavily rely on CNN-based architectures. Alternatively, we rethink this task from a convolution-free sequence-to-sequence perspective and predict saliency by modeling long-range dependencies, which can not be achieved by convolution. Specifically, we develop a novel unified model based on a pure transformer, namely, Visual Saliency Transformer (VST), for both RGB and RGB-D salient object detection (SOD). It takes image patches as inputs and leverages the transformer to propagate global contexts among image patches. Unlike conventional architectures used in Vision Transformer (ViT), we leverage multi-level token fusion and propose a new token upsampling method under the transformer framework to get high-resolution detection results. We also develop a token-based multi-task decoder to simultaneously perform saliency and boundary detection by introducing task-related tokens and a novel patch-task-attention mechanism. Experimental results show that our model outperforms existing methods on both RGB and RGB-D SOD benchmark datasets. Most importantly, our whole framework not only provides a new perspective for the SOD field but also shows a new paradigm for transformer-based dense prediction models. Code is available at https://github.com/nnizhang/VST.
Alex: Their encoder is a pretrained model called T2T-ViT, short for tokens-to-token. A token here is just one image patch represented as a vector. The tokens-to-token step reshapes the sequence back into a little image. Then it gathers each small, overlapping neighbourhood into a single new token. The sequence gets shorter at every step. The authors simply run that in reverse. Each token is expanded into a small patch of sub-tokens, overlapping patches are folded back together, and the sequence gets longer.
Sam: And how many of those reverse steps does it take to get back to full size?
Alex: Three. From one-sixteenth scale they go up in stages to full resolution. Along the way they splice in the early, high-resolution tokens from the encoder, which carry fine local structure. That's the multi-level fusion trick convolutional models use, rebuilt in token form.
Sam: Couldn't you just use ordinary bilinear upsampling on the tokens and save yourself the trouble?
Alex: They tested exactly that. On the NJUD depth dataset they score with a structure measure, which rates how well the predicted shape matches the true object. The baseline, which predicts straight from the coarse tokens, scores about 0.87. Adding bilinear upsampling lifts it to about 0.91. Swapping in their reverse step adds roughly another point. So most of the jump comes from having resolution at all. Their method adds a smaller but consistent gain over bilinear on all four datasets they used for this test.
Sam: Now the second problem. How do tokens become a saliency map without convolution?
Alex: This is the idea I'd take away. Image classifiers add a special class token and read the answer off it. But one token gives you one answer, not a map. So the authors add two learned tokens: a saliency token and a boundary token. These ride along with the patch tokens through the decoder's transformer layers, picking up what this particular image looks like. At the end comes what they call patch-task-attention. Every patch token is compared against the saliency token, and that score decides how much of the saliency token's content gets added to that patch. Do the same with the boundary token, upsample once more, and you get two maps.
Sam: So rather than a separate decoder branch per task, you just add a token per task.
Alex: Yes, and the cost difference is measurable. They compared it with a conventional design that runs two parallel decoders. The token version needs about 18 billion multiply-accumulate operations against about 28 billion for the two-decoder version. It also uses fewer parameters, and it scored better on three of four datasets.
Sam: Three of four. What happened on the fourth?
Alex: Essentially a tie, a hair lower. The same pattern shows up for adding boundary detection at all: it helps on three datasets and not on NJUD. These are small effects.
Sam: Let's get to the headline comparison. Against what, and how much?
Alex: For the depth version, fourteen published methods across nine benchmarks. For plain colour images, twelve methods across six benchmarks. They score with four standard metrics, including the structure measure and mean absolute error, which is the average per-pixel mistake. The clearest win is on the DUTLF-Depth dataset. There the structure measure is 0.943, against 0.923 for the next best method.
Sam: Twenty thousandths is a clear win in this field?
Alex: On that dataset, by the standard of the table, yes. Most other margins are a point or less. And I'd push back on the summary that it beats everything. On several datasets, including DUT-O and DUTS, another method has lower mean absolute error. On the structure measure and the related scores, it leads almost across the board.
Sam: What about cost? Transformers aren't known for being cheap.
Alex: The authors say comparable, and broadly that holds. They chose their backbone because it's similar in compute to ResNet-50. The depth model uses about 31 billion operations, lower than many rivals. But it has about 84 million parameters, which is more than most of the depth methods it's compared against. The colour-only model is lighter.
Sam: What would make you hesitate before building on this?
Alex: Three things, all from the paper's own setup. The margins are often in the second or third decimal place, and no variation across training runs is reported. So I can't say how many gains would survive reseeding. The component tests were run only on the depth model, though the colour model shares most parts. And "pure transformer" deserves a footnote. The overlapping patch splitting deliberately builds in local neighbourhood structure, which is the kind of prior convolution provides.
Sam: So who should actually sit down with the full paper, and where should they start?
Alex: Anyone building transformer models for segmentation or other per-pixel tasks. Start with the multi-task decoder section, where the task tokens and patch-task-attention are defined. Then read the ablation study, which separates the upsampling, fusion, and decoder gains. Saliency specialists should also read the cross-modality transformer for fusing colour and depth, and the full comparison tables.
Sam: And for everyone else, the one line to carry around?
Alex: To get a map from a transformer, give it a token for the task. Then ask every patch how much it agrees.
Sam: Small margins, but a clean idea worth borrowing.