DenseOn and LateOn: Fully Open Dense and Late-Interaction Retrieval Models Set New BEIR SOTA

Paper Overview

Field: NLP (Information Retrieval) Authors: Raphaël Sourty, Antoine Chaffin, Paulo Roberto Moura Junior, Amélie Chatelain Published: 2026-07-29 arXiv: 2607.27178

Summary

State-of-the-art retrieval models increasingly rely on closed training data, creating a reproducibility gap. This paper presents an open end-to-end recipe for training retrieval models and studies how English supervision transfers to multilingual retrieval through translate-train.

Key Contributions

• Data curation: Reconstructed and curated 665M English contrastive pre-training pairs from 1.4B pairs across 34 public sources, plus 1.88M supervised fine-tuning pairs with mined hard negatives.
• Two open English models (149M parameters each):
• DenseOn — a single-vector dense model, achieving 56.20 average nDCG@10 on BEIR.
• LateOn — a ColBERT-style late-interaction model, achieving 57.22 average nDCG@10 on BEIR.
• Both set new state-of-the-art results for this size class.
• Multilingual scaling: The validated English data was translated into eight languages, yielding 2.8B pairs (including cross-lingual samples) used to train mDenseOn and mLateOn, two 307M-parameter models based on mmBERT-base.

Key Finding

Despite sharing the same backbone, data, and training objectives, the two model families exhibit different representation behavior:

• The dense model is strong on English and translated languages but degrades outside the translate-train coverage.
• The late-interaction model generalizes better to unseen languages and scripts.

This suggests that token-level matching turns translate-train from a target-language expansion strategy into a multilingual generalization recipe.

The authors release models, datasets, and training code openly.

--- *Auto-collected on 2026-07-31*