CUST : Clustered Unit-level Similarity Transformer for Lightweight Image Super-Resolution

Single Image Super-Resolution (SISR) aims to reconstruct high-resolution images from low-resolution inputs. While Vision Transformers (ViTs) have shown great promise in this task, their quadratic computational complexity makes them difficult to deploy on resource-constrained hardware. Existing solutions, such as window-based attention, often sacrifice the long-range dependency modeling that gives ViTs their power. This paper asks: can we design a transformer architecture that maintains the global context of ViTs while achieving the efficiency required for real-world, lightweight deployment?

The author introduces the Clustered Unit-level Similarity Transformer (CUST), which balances global and local information through two primary modules:

1. Cross-window Affinity Neighbor Attention (CANA): Instead of restricting attention to fixed local windows, CANA groups semantically similar patches across the entire image. It uses representative window tokens to cluster related patches, allowing the model to attend to relevant information outside the immediate local neighborhood.
2. Multi-frequency Error-driven Dense Attention (MEDA): This module focuses on local fidelity. It calculates the residual difference between the original features and their downsampled-upsampled versions to isolate high-frequency details (like edges). These error signals are then refined via dilated convolutions and used to guide the attention mechanism, ensuring the model prioritizes the reconstruction of fine structures.

CUST demonstrates a superior trade-off between restoration performance and computational cost. Compared to state-of-the-art window-based attention models, CUST-Base achieves significantly broader spatial context while reducing average inference latency by 26.7%. Furthermore, it provides substantial memory savings compared to existing token-generation methods, making it highly suitable for resource-constrained environments without compromising on image quality metrics like PSNR.