Spectral Heat Flow for Conservative Token Condensation in Vision-Language Models

Vision-Language Models (VLMs) are computationally expensive during inference, largely due to the high number of visual tokens processed by the language decoder. Existing token pruning methods often rely on simple attention-based ranking, which can lead to "spatial holes" and the loss of critical contextual information when high compression ratios are applied. This paper asks: can we develop a training-free pruning framework that preserves both the structural integrity of the image and the necessary statistical diversity of the visual tokens?

SpecFlow introduces a three-step "conservative condensation" paradigm to replace destructive pruning:

1. Spectral Heat Flow: Instead of relying on raw, potentially noisy attention scores, SpecFlow treats visual tokens as nodes in a k-nearest neighbor (kNN) graph. It propagates initial saliency (derived from the [CLS] token) using spectral heat diffusion, which smooths the importance field and ensures that importance is distributed coherently across regions.
2. Adaptive Spatial Partitioning: To prevent the model from ignoring entire regions of an image, SpecFlow uses a quadtree-based approach to partition the image. It allocates token budgets proportionally to the energy mass of each region, ensuring that both salient foreground objects and informative background context are represented.
3. Coreset Sink Aggregation: Rather than simply discarding tokens, SpecFlow compresses the pruned information into "sink tokens"—a mean representation and a residual token—which are passed to the decoder to preserve global context and diversity.

SpecFlow is plug-and-play, requires no fine-tuning, and is compatible with standard optimizations like FlashAttention. By shifting from point-wise pruning to structure-aware condensation, the authors demonstrate that LLaVA-1.5 can retain 95.6% of its original performance while discarding nearly 89% of its visual tokens. This provides a robust, efficient solution for deploying large-scale multimodal models on resource-constrained hardware.