DenseOn with the LateOn: Fully Open Dense and Late-Interaction Models for Multilingual, Long-Context, and Code Search

This paper addresses the reproducibility gap in modern retrieval models by providing a fully open, end-to-end training recipe. The authors curate a massive English retrieval dataset from 34 public sources, applying a non-destructive filtering pipeline that allows users to adjust quality thresholds. They train two 149M-parameter models—DenseOn (a single-vector dense model) and LateOn (a ColBERT-style late-interaction model)—to establish a baseline. To extend these to multilingual, long-context, and code retrieval, the authors use a translate-train strategy, creating a 2.8B-pair multilingual corpus by translating the English data into eight target languages. They then train 307M-parameter versions, mDenseOn and mLateOn, using the mmBERT-base backbone.

• Superiority of Late Interaction: While both dense and late-interaction models perform well on English and translated languages, the late-interaction architecture (mLateOn) demonstrates significantly better generalization to languages and scripts that were never seen during training.
• Translate-Train as Generalization: The authors argue that for late-interaction models, translate-train functions as a multilingual generalization recipe rather than just a target-language expansion strategy, likely because token-level matching preserves cross-lingual alignment more effectively than dense vector pooling.
• State-of-the-Art Performance: Both DenseOn and LateOn set new state-of-the-art results for their size class on the BEIR benchmark, and their multilingual counterparts show strong performance across MIRACL, MLDR, and MTEB Code benchmarks.

This work provides the research community with a transparent, open-source alternative to closed-source retrieval systems. By releasing the full training data, filtering metadata, and code, the authors enable controlled experiments that isolate the impact of data quality and training recipes on retrieval performance. The finding that late-interaction models are more robust to unseen languages provides a clear architectural recommendation for developers building multilingual search systems with limited training resources.