Stellar: Scalable Multimodal Document Retrieval for Natural Language Queries

Multimodal document retrieval is critical for RAG systems, but current state-of-the-art methods—which use multi-vector representations and late interaction—are notoriously memory-intensive. The authors investigate how to maintain the high retrieval effectiveness of these multi-vector models while overcoming the massive memory and storage bottlenecks that hinder their deployment at scale.

The authors propose Stellar, a framework that shifts from an all-in-memory paradigm to a disk-backed architecture. Stellar operates in two stages:

1. Lexical Representation-based Filtering (LRF): The system fine-tunes an MLLM to act as a sparse encoder, projecting both documents and queries into a high-dimensional sparse lexical space. This allows for rapid, memory-efficient candidate filtering using an inverted index.
2. Efficient Disk-backed Late Interaction (DLI): Instead of keeping all token-level embeddings in RAM, Stellar stores them on disk. It uses a balanced clustering algorithm to group semantically related documents into disk blocks. A cost-aware loading strategy then dynamically fetches only the necessary embeddings into memory during the retrieval process.

Stellar achieves a significant reduction in resource consumption without sacrificing retrieval accuracy. By filtering the document corpus down to a manageable candidate set before performing the expensive late interaction, the system reduces memory usage and query latency by 1-2 orders of magnitude compared to existing multi-vector baselines like ColPali. The authors also introduce "LargeDoc," a new large-scale benchmark dataset, to validate the framework's performance in real-world scenarios.

As RAG systems increasingly incorporate complex multimodal documents (e.g., charts, diagrams, and reports), the scalability of the retrieval component becomes a primary constraint. Stellar provides a practical path for deploying high-precision, multi-vector retrieval systems on standard hardware, significantly lowering the barrier to entry for large-scale multimodal information retrieval.