Capturing Token Tendencies for Training-Free Token Pruning in Multimodal Large Language Models

Multimodal Large Language Models (MLLMs) have become essential for complex reasoning tasks, but their reliance on high-resolution visual inputs leads to a massive number of visual tokens. Because the standard attention mechanism has quadratic complexity, this creates a significant computational bottleneck. Existing solutions typically use static pruning—discarding tokens based on a single-layer importance score—which is inherently irreversible. This paper argues that such methods fail because token importance is not static; it evolves as the model processes information from shallow, texture-focused layers to deep, semantic-focused layers.

Instead of treating pruning as a snapshot decision, the authors reformulate it as a temporal trajectory modeling problem. The core of their approach, Trend-aware Pruning, consists of three main components:

• Layer-wise Token Collector: Maintains a sliding window of historical attention scores to track how the importance of each token changes as it moves through the decoder layers.
• Adaptive Flow Identification: Categorizes tokens based on their trajectory patterns—specifically identifying 'upward-trending' tokens (those gaining importance), 'downward-trending' tokens (those losing relevance), and 'fluctuating' tokens (those with volatile importance).
• Flow Activation: A dynamic mechanism that selectively 'rescues' or reactivates tokens that were initially undervalued but show rising semantic importance, ensuring that critical visual cues are not permanently lost during the pruning process.

By moving away from static, layer-isolated pruning, this method allows MLLMs to maintain high performance while significantly reducing the number of visual tokens processed. Experimental results show that the approach can reduce visual tokens by over 77.8% while retaining competitive accuracy across diverse benchmarks like MMB, GQA, and POPE. This provides a robust, training-free, and reversible solution for high-efficiency multimodal inference that adapts to the specific needs of the input task.