Fusion Embedding: A Unified Embedding Space for Text, Image, Video, and Audio

How can a pre-trained, frozen vision-language embedding model be extended to include audio as a first-class modality without altering its existing performance on text, image, and video retrieval? The authors seek to create a unified embedding space that allows a single index to serve queries across all four modalities while maintaining the integrity of the original backbone.

The authors introduce the Fusion Embedding family, built on the frozen Qwen3-VL-Embedding-2B base and the Qwen2.5-Omni audio tower. They propose two generations of improvements:

• Generation 1 (fusion-embedding-1): Uses a 16.4M-parameter 'FusionResampler' connector to map audio features into the base model's input space. This allows the frozen LLM to process audio tokens alongside other modalities.
• Generation 2 (fusion-embedding-2): Adds 44.2M-parameter bottleneck adapters to the decoder layers. These are 'modality-gated,' meaning they only activate when audio is present. For text, image, or video inputs, the adapter branch is bypassed, ensuring the output remains bit-for-bit identical to the original base model.

Training utilizes a symmetric InfoNCE loss with a frozen-text memory bank, which allows for efficient training on a single GPU by avoiding the need to recompute text embeddings during the training loop.

The study demonstrates that aligning audio to text within a shared space allows audio-to-image retrieval to emerge naturally, even without any paired audio-visual training data. The authors report that their approach achieves competitive retrieval performance while providing a 'bitwise invariance' guarantee—a property where non-audio inputs are processed exactly as they would be by the original, unmodified base model. They also provide controlled negative results, showing that certain design choices (like widening the connector or using LLM-rewritten captions) can actually degrade performance, providing a roadmap for future frozen-backbone research.