DenseOn with the LateOn: Fully Open Dense and Late-Interaction Retrieval Models with New BEIR SOTA at 149M Parameters

Paper Overview

Field: NLP / Information Retrieval Authors: Raphaël Sourty, Antoine Chaffin, Paulo Roberto Moura Junior, Amélie Chatelain Posted: 2026-07-29 arXiv: 2607.27178

Summary

State-of-the-art retrieval models increasingly rely on closed training data, creating a reproducibility gap. This work presents an open end-to-end recipe for training retrieval models and studies how English supervision transfers to multilingual retrieval through translate-train.

Key Contributions

• Data curation: Reconstructed and curated 665M English contrastive pre-training pairs from 1.4B pairs across 34 public sources; built 1.88M supervised fine-tuning pairs with mined hard negatives.
• Two 149M-parameter models:
• DenseOn — single-vector dense model
• LateOn — ColBERT-style late-interaction model
• Results: 56.20 (DenseOn) and 57.22 (LateOn) average nDCG@10 on BEIR — new state-of-the-art for this size class.

Multilingual Extension

The validated English data was translated into eight languages, producing 2.8B pairs (including cross-lingual samples) used to train mDenseOn and mLateOn, two 307M-parameter models based on mmBERT-base.

Despite sharing the same backbone, data, and objectives, the models behave differently:

• The dense model is strong on English and translated languages but degrades outside translate-train coverage.
• The late-interaction model generalizes better to unseen languages and scripts.

This suggests token-level matching turns translate-train from a target-language expansion strategy into a multilingual generalization recipe.

Models, datasets, and training code are publicly released.

Original Abstract (excerpt)

> State-of-the-art retrieval models increasingly rely on closed training data, creating a reproducibility gap. We present an open end-to-end recipe for training retrieval models and study how English supervision transfers to multilingual retrieval through translate-train. We first reconstruct and curate 665M English contrastive pre-training pairs from 1.4B pairs across 34 public sources and build 1.88M supervised fine-tuning pairs with mined hard negatives. Training yields two 149M-parameter models: DenseOn, a single-vector dense model, and LateOn, a ColBERT-style late-interaction model. They achieve 56.20 and 57.22 average nDCG@10 on BEIR, respectively, setting new state-of-the-art results for this size class. We then translate the validated English data into eight languages, yielding 2.8B ...

*Auto-collected on 2026-07-31*