UEmbed: Unified Sparse and Dense Multimodal Embeddings

Modern retrieval systems often rely on either dense embeddings for semantic matching or sparse lexical methods (like BM25 or SPLADE) for interpretability and exact term matching. Existing learned sparse retrieval (LSR) methods typically require encoder-style bidirectional architectures, which complicates their integration into modern decoder-only Large Language Models (LLMs) and limits their native multimodal capabilities. This paper asks: can we unify dense and sparse retrieval within a single, decoder-only multimodal backbone without auxiliary cross-modal modules?

To overcome the architectural constraints of causal models, the authors introduce UEmbed. Instead of using a single token to project into the entire vocabulary space—which creates an information bottleneck—UEmbed appends $N$ learnable special tokens to the input sequence. The vocabulary is partitioned into $N$ disjoint subsets using k-means clustering, and each special token is responsible for predicting sparse weights over its assigned subset. The final sparse vector is the concatenation of these subset projections. Dense embeddings are simultaneously derived from the hidden state of the EOS token. The model is trained using a unified objective that combines InfoNCE losses for both dense and sparse modes, along with FLOPS regularizers to encourage sparsity.

UEmbed demonstrates that a single decoder-only backbone can effectively support both dense and sparse retrieval across text and multimodal inputs. On the MMEB-v2 benchmark, UEmbed-9B achieves state-of-the-art results for sparse multimodal retrieval and outperforms other publicly available multimodal embedding models in dense retrieval. The sparse representation remains highly competitive with the dense mode, with performance gaps of less than 1.0 point across tested scales. Furthermore, the model remains competitive on the BEIR text-retrieval benchmark, matching specialized models like Echo-Mistral-SPLADE while maintaining compatibility with high-throughput serving stacks like vLLM.