Spectral Evolution-Guided Token Pruning in Multimodal Large Language Models

Multimodal Large Language Models (MLLMs) often suffer from high computational costs due to processing large numbers of visual tokens. Existing token pruning methods typically rely on instantaneous, single-layer signals like attention scores or feature magnitudes. These methods often suffer from positional bias and fail to capture the dynamic transformation of visual information as it moves through the deep layers of a Transformer. The authors ask: can we identify redundant tokens more reliably by analyzing how visual representations evolve across layers?

The authors propose Cross-Layer Spectral Evolution (CLSE), a training-free framework that treats token importance as a function of its spectral dynamics. Instead of looking at a single layer, CLSE transforms visual tokens into the frequency domain and applies a Gaussian high-pass filter to isolate high-frequency structural information. By measuring the change in this spectral energy between adjacent layers, the model identifies tokens that undergo significant 'detail-to-semantics' evolution. Tokens that show stronger spectral redistribution are prioritized for retention, while those that remain relatively static are pruned. This approach is computationally efficient, requires no retraining, and is orthogonal to existing token-merging techniques.

CLSE consistently outperforms existing state-of-the-art pruning methods across a wide range of image and video benchmarks. In experiments with models like LLaVA-1.5 and Qwen2-VL, CLSE maintains higher accuracy under aggressive token reduction (e.g., pruning up to 88.9% of tokens) compared to attention-based or magnitude-based baselines. The authors demonstrate that by focusing on cross-layer spectral dynamics, the model effectively mitigates the positional bias inherent in standard attention mechanisms, leading to a superior trade-off between inference latency and reasoning performance.