Suhyeong Park, Junha Jung, Jungwoo Park, Jaewoo Kang
6 min
Multi-vector vision-language retrieval models, such as ColPali, achieve high accuracy by performing late-interaction (MaxSim) matching between query tokens and dense image-side patch embeddings. However, storing and scoring these dense representations is computationally expensive. Existing compression methods often rely on simple pruning or feature-based pooling, which can inadvertently discard or collapse critical object-level information needed for specific queries. SaMer (Semantic-aware Merging) addresses this by introducing an object-aware token merging framework that compresses visual tokens while ensuring that the resulting representatives remain selectable for future queries.
SaMer compresses post-projector visual tokens into representative centroids using a feature-spatial soft assignment mechanism. Unlike standard pooling, SaMer uses object annotations during training as a 'merge prior' to discourage the merging of tokens from different object instances. This ensures that the compressed representatives retain distinct semantic evidence. The framework adapts only the shared projection layer, keeping the vision and language backbones frozen, and requires no ground-truth bounding boxes or detectors during inference. This allows SaMer to be plugged into existing multi-vector retrieval architectures with minimal overhead.
By reducing the number of image-side tokens by over 93% (at ), SaMer achieves a 16.09x reduction in storage requirements for models like ColPali. Crucially, it demonstrates that efficient retrieval is not merely a matter of reducing token counts, but of preserving the specific visual evidence that query tokens need to select. The empirical results show that SaMer outperforms existing pruning and pooling baselines on standard benchmarks like Flickr30K and MSCOCO, proving that object-aware compression is superior for maintaining phrase-level grounding and retrieval accuracy.
Multi-vector vision-language retrieval preserves fine-grained visual evidence through maximum-similarity late interaction, but dense image-side tokens make storage and scoring expensive. Existing token compression methods reduce this cost, yet they can remove or collapse object- and region-level evidence that future query tokens may need to select. We propose SaMer, an object-aware token merging framework that compresses image-side post-projector tokens into $K$ representative centroids while preserving the original late-interaction interface. SaMer uses object annotations only during training as a merge prior to discourage cross-instance mixing, requires no ground-truth bounding boxes or detectors at inference time, and adapts only the shared projection layer with frozen vision and language backbones. With $K=64$, SaMer removes more than 93% of image-side tokens and reduces ColPali storage by $16.09\times$, while improving R@1 on Flickr30K and MSCOCO. These gains arise because object-aware merging preserves query-selectable object evidence that pruning or feature-only pooling can remove or collapse. SaMer also outperforms compression baselines and shows stronger phrase-level grounding, suggesting that efficient multi-vector retrieval depends not only on reducing token count, but on preserving the evidence future query tokens need to select.
Alex: So it's checking both what something looks like and where it sits in the photo before deciding whether to group it with something else?
Sam: Precisely. That combination — visual similarity plus spatial position — is what they call feature-spatial soft assignment. It's what keeps the evidence anchored to specific regions, so the system doesn't accidentally blend the sky into someone's collar.
Alex: And once the fragments are sorted into these smarter bins, how does the actual searching work?
Sam: The matching process — which the researchers call late interaction, using a technique called MaxSim — compares your search words against these bins at the very end of the process. The important thing is that SaMer doesn't break that matching step. It just feeds it fewer, better-organized bins instead of hundreds of raw fragments. The search engine still does the same kind of careful comparison; it just has cleaner, more structured evidence to work with.
Alex: What does that mean in practice for storage?
Sam: The paper reports removing more than ninety-three percent of the visual tokens, which reduced the storage needed for the index by more than sixteen times. And critically, the search accuracy didn't drop — it actually improved slightly. The suggestion is that a lot of the raw data was redundant noise, and by keeping only the object-aware bins, the system had sharper evidence to match against.
Alex: That's a notable result. Less data, but better answers.
Sam: It points to something the paper argues more broadly: the bottleneck for these systems isn't raw data volume — it's how intelligently you index it. If you prune data based purely on which fragments look "important" in isolation, you often discard the specific details a user will search for later. A fragment describing a small red umbrella might not look significant on its own, but it becomes critical the moment someone searches for exactly that.
Alex: Did they test this on anything other than natural photographs?
Sam: They did. They tested SaMer on document datasets, where retrieval depends more on layout — the position of headings, tables, and text blocks — than on recognizable objects. Even in that context, the core logic held up. Keeping spatially distinct regions separate still helped the system retrieve the right documents. The researchers treat that as evidence the approach is reasonably general, not just a solution tuned for one type of image.
Alex: So the underlying principle — respect the boundaries between distinct things — turns out to be useful beyond just photos of people and umbrellas.
Sam: That seems to be the paper's broader claim. And there's one more detail worth noting. To make this work, the researchers didn't retrain the entire vision system from scratch. They froze the main encoders — the large pre-trained models that understand images and language — and only trained a smaller connection layer. That layer learns to produce features that stay useful even after being compressed into bins. The system is essentially learning, at a structural level, how to be compact without becoming uninformative.
Alex: It's adapting to its own constraints.
Sam: Exactly. Rather than compressing a finished representation and hoping for the best, the model learns to build representations that are already suited to compression. That's a meaningful architectural choice, and it's part of why the accuracy holds up even after such aggressive reduction.
Alex: So the headline finding is that intelligent, object-aware compression can outperform brute-force storage — and the key is teaching the system to respect the boundaries between things before it ever runs a search.
Sam: That's a fair summary. The work suggests that for large-scale image retrieval, how you organize your index matters at least as much as how much data you keep. Thanks for listening to ResearchPod.