KaLM-Reranker-V1: Fast but Not Late Interaction for Compressed Document Reranking

As retrieval systems scale, traditional rerankers—which jointly encode queries and passages—become computationally prohibitive due to the need to re-process the entire input for every query. This paper addresses the trade-off between the high accuracy of joint-encoding models and the high efficiency of late-interaction models by proposing a new architecture that decouples these processes without sacrificing interaction depth.

The authors introduce KaLM-Reranker-V1, an encoder-decoder model that separates passage encoding from query-aware scoring. The encoder processes passages offline, generating compressed representations using Matryoshka embedding pooling (MEP). During inference, the decoder takes the query and instruction, then uses cross-attention to interact with the pre-computed passage representations. This FBNL (Fast but Not Late-interaction) design allows the system to reuse passage embeddings while still capturing fine-grained relevance signals that standard late-interaction models often miss.

KaLM-Reranker-V1 demonstrates that it is possible to achieve state-of-the-art reranking performance with significantly lower online latency. On the BEIR benchmark, the model performs on par with industrial-grade rerankers while being substantially faster. The use of Matryoshka embedding pooling allows for flexible storage-performance trade-offs, where moderate compression (e.g., 2x to 4x) results in negligible accuracy loss. Furthermore, even the smallest 0.27B parameter model remains highly competitive, outperforming much larger embedding models in long-horizon memory retrieval tasks.

This work provides a practical path for deploying high-quality reranking in large-scale production environments. By enabling offline pre-computation of document representations, the model removes the primary bottleneck of modern reranking pipelines, allowing developers to integrate sophisticated, instruction-aware relevance modeling into systems that were previously limited to simpler, less accurate retrieval methods.