Vision-language models (VLMs) project images into hundreds to thousands of visual tokens, making decoder inference expensive in both attention computation and KV-cache memory. Existing visual-token reduction methods largely follow a rank-and-remove paradigm: they score visual tokens, keep a compact subset, and permanently discard the rest. We show that this irreversible action is fragile because visual-token importance changes across decoder depth; tokens ranked low at one stage may become relevant in later layers, especially for grounding-sensitive queries. We propose Reroute, a training-free plug-in that replaces removal with recoverable routing. At each routing stage, selected vision tokens pass through decoder blocks, while deferred tokens bypass the stage and re-enter the candidate pool at the next routing decision. Reroute reuses existing attention-score ranking rules and stage-wise schedules, preserving the theoretical TFLOPs and KV-cache budget class of the pruning method it augments. Across FastV, PDrop, and Nüwa variants on LLaVA-1.5 and Qwen backbones, reroute improves grounding under aggressive token reduction while maintaining general VQA performance. These results suggest that VLM token reduction should not be viewed only as irreversible pruning, but also as recoverable routing. The code can be found here: https://github.com/elmma/mllm-reroute/
Alex: Welcome to another episode of ResearchPod. Today, we're looking at a paper titled "Reroute, Don't Remove." Sam, what is this study actually about?
Sam: It's about making a particular kind of AI model faster and cheaper to run—without breaking it in the process. These models, called vision-language models, can look at an image and answer questions about it. But to do that, they first have to chop the image up into hundreds of small pieces of information called "tokens." Processing all those tokens takes a lot of computing power, so researchers are always looking for ways to cut that number down.
Alex: And the problem is that the current ways of cutting them down are causing problems?
Sam: Exactly. The standard approach is what the authors call "rank-and-remove." The model looks at all its tokens, gives each one a score for how important it seems, and then permanently deletes the low-scoring ones. The authors argue this is a mistake. A token that looks unimportant early on—say, a patch of plain blue sky in a photo—might turn out to be essential later when the model is trying to understand the full scene.
Alex: So the core problem is that once a token is deleted, it's gone forever. There's no way to recover it.
Sam: That's the heart of it. Think of it like hospital triage. If a patient is sent home because they don't look seriously ill at the front desk, the hospital can't help them if their condition gets worse an hour later. The authors are saying: don't send those borderline cases home. Keep them in a waiting room.
Alex: That makes sense as a principle. But if you're keeping all those extra tokens around, doesn't that defeat the whole point of trying to make the model faster?
Sam: That's the key question, and it's where the paper gets interesting. Their solution is called "Reroute." Instead of deleting a low-scoring token, the model lets it skip the current layer's heavy processing—think of it as stepping aside in the queue—but it stays in the sequence. If a later layer decides that token is actually relevant, it can be pulled back in and processed then.
Alex: So the model gets a second chance to use information it initially set aside?
Sam: Right. And because only the tokens that are actively needed get sent through the computationally expensive steps at any given moment, the total processing cost stays roughly the same as the old delete-everything approach. You're not paying extra to keep those tokens waiting. They're just parked, not processed.
Alex: It's like a smart queue rather than a shredder. How does the model decide which tokens go into that waiting area in the first place?
Sam: That's one of the more practical aspects of the design. The model already produces ranking scores as part of its normal operation—it's part of how the attention mechanism works, which is the process by which the model figures out which parts of the image to focus on. The authors just reuse those scores to decide what gets rerouted. There's no extra training required. You can take an existing model—they tested it on systems like LLaVA and Qwen—and plug this method straight in.
Alex: And the results hold up?
Sam: The results are consistent across the models they tested. The area where the improvement was most visible was in what's called "grounding"—the model's ability to correctly locate and point to a specific object in an image when asked about it. That task is quite sensitive to losing visual information, so it's a good stress test. Under the most aggressive efficiency settings, the standard deletion methods would often fail badly, while the Reroute approach stayed stable.
Alex: So it's less about raw speed and more about preserving the quality of understanding even as you push the model harder.
Sam: That's a good way to put it. The deeper point the paper is making is that information in these networks doesn't have a fixed value. Something that seems irrelevant at step three might be exactly what the model needs at step seven. Permanent deletion assumes you can know the future importance of a token from an early snapshot—and that assumption, the authors suggest, is often wrong. Rerouting is a way of staying humble about what you don't yet know.
Alex: That's a useful principle well beyond AI, honestly. Thanks for walking us through it, Sam, and thanks to everyone listening to ResearchPod.