Accelerating Multimodal Large Language Models with Prior-Corrected Token Reduction

Multimodal Large Language Models (MLLMs) often process images as long sequences of visual tokens, leading to high computational costs during inference. While existing training-free pruning methods use attention scores to discard redundant tokens, these methods often fail because attention is biased by a model-induced prior—a tendency for the model to focus on visually prominent regions regardless of the actual task instruction. This paper investigates how to distinguish between these task-agnostic priors and true instruction-conditioned semantic relevance to improve token pruning.

The authors propose Prior-Corrected Token Reduction (PriorTR), a training-free method that explicitly separates the model-induced prior from the task-conditioned posterior. To achieve this without the overhead of multiple forward passes, the authors introduce a null token (e.g., a separator token) placed after the visual tokens. Because the decoder is causal, this null token cannot attend to the instruction, making its attention distribution a natural estimate of the model's instruction-agnostic prior. By contrasting this prior with the posterior attention from the instruction tokens, the model calculates a 'semantic lift' for each visual token. Tokens are then ranked and pruned based on this lift, ensuring that only those contributing unique, task-relevant information are retained.

PriorTR consistently outperforms existing training-free baselines across multiple MLLMs and benchmarks, particularly under aggressive token budgets where traditional methods often discard critical visual evidence. By physically pruning the hidden states and KV cache at an intermediate layer, the method provides tangible speedups and memory savings. The authors demonstrate that their approach effectively mitigates 'prior-dominated masking,' where background noise is erroneously preserved over task-relevant features. The method is plug-and-play, requiring no additional training or architectural changes.