Out of Sight, Still in Mind: Token Compression for Omni-LLMs

Omni-modal large language models (Omni-LLMs) process audio, video, and text simultaneously, but the visual stream creates a massive, redundant token burden that leads to high latency and memory costs due to the quadratic complexity of attention. This paper asks whether it is possible to reduce this visual token count without training a new model or sacrificing reasoning accuracy.

The authors propose ReMo, a training-free framework that treats token compression as a cross-modal information redistribution problem. Instead of simply pruning tokens, ReMo uses three main strategies:

1. Cross-modal projection: It aligns audio and visual embeddings into a common canonical space. This allows the model to identify and discard visual tokens that are already "explained" by the accompanying audio or other visual context.
2. Semantic offloading: It uses a lightweight object detector to identify objects in the video. These objects are converted into text labels (proxies) and injected into the input sequence. By assigning these proxies specific temporal and spatial positions using TMRoPE (Temporal-Multi-Resolution Rotary Positional Embeddings), the model retains the object's meaning and location while using significantly fewer tokens than raw visual patches.
3. Detail restoration and merging: It restores high-variance local visual details that might have been discarded by global saliency scores and merges temporally redundant tokens across adjacent video chunks.

ReMo was evaluated on Qwen2.5-Omni models at 3B and 7B scales across five audio-visual benchmarks. The framework successfully removed 54% of input tokens. Remarkably, it did not just maintain performance; it slightly exceeded the full-token model's accuracy, reaching 101.2% and 101.3% of the baseline average. This suggests that offloading visual semantics to text and audio is a highly effective way to compress multimodal inputs without losing critical information.