SlimVLM: Sensitivity-aware Dynamic Structured Pruning with Adaptive Visual Token Selection for Efficient Vision-Language Models

Vision-Language Models (VLMs) are computationally expensive due to their large parameter counts and the high number of visual tokens generated by image encoders. While pruning is a standard technique for compressing Large Language Models (LLMs), applying it directly to VLMs is difficult because redundant visual tokens distort the estimation of which model components are truly important. This paper asks how to effectively prune VLMs while maintaining task performance and computational efficiency.

The authors propose SlimVLM, a structured pruning framework consisting of two primary innovations. First, they introduce an adaptive visual token selection strategy that uses text-to-visual attention maps to identify and remove redundant visual tokens before calculating module importance. This ensures that the pruning process focuses on the most semantically relevant information. Second, they implement a sensitivity-aware dynamic pruning mechanism. This mechanism calculates the linear reconstruction error between pruned and unpruned modules using the Pearson correlation coefficient, allowing the model to assign different pruning ratios to different modules based on their individual tolerance for sparsity.

SlimVLM consistently outperforms existing structured pruning baselines (such as FLAP and Wanda-sp) across multiple multimodal benchmarks, including LLaVA-1.5 and LLaVA-Next. At a 20% pruning ratio, SlimVLM retains near-original accuracy, achieving state-of-the-art performance for compressed VLMs. The authors also demonstrate that the performance of pruned models can be further recovered through LoRA fine-tuning, making the framework a practical solution for deploying VLMs on resource-constrained devices.

As VLMs become essential for tasks like visual question answering and embodied AI, their deployment on edge devices is hindered by memory and compute constraints. SlimVLM provides a robust, structured way to reduce model size without the performance degradation typically associated with naive pruning, offering a clear path toward more efficient multimodal AI deployment.