[论文] DenseOn with the LateOn: Fully Open Dense and Late-Interaction Models ...

论文概要

研究领域: NLP 作者: Raphaël Sourty, Antoine Chaffin, Paulo Roberto Moura Junior, Amélie Chatelain 发布时间: 2026-07-29 arXiv: 2607.27178

中文摘要

最先进的检索模型越来越依赖封闭的训练数据，造成了可复现性差距。本文提出一个开放的端到端检索模型训练方案，并研究英语监督如何通过translate-train迁移到多语言检索。我们首先从34个公共来源的14亿对数据中重构和精选6.65亿英语对比预训练对，并构建188万带有挖掘难负例的监督微调对。训练得到两个149M参数模型：DenseOn（单向量稠密模型）和LateOn（ColBERT风格的迟交互模型）。它们在BEIR上分别取得56.20和57.22的平均nDCG@10，为该规模类别创下新SOTA。然后我们将验证后的英语数据翻译成八种语言，产生28亿对含跨语言样本的数据，训练mDenseOn和mLateOn——两个基于mmBERT-base的307M参数模型。尽管共享主干网络、数据和目标，它们的表征行为却不同：稠密模型在英语和翻译语言上很强，但在translate-train支持范围外退化；而迟交互模型对未见过语言和文字的泛化更好。这表明token级匹配将translate-train从目标语言扩展策略转变为多语言泛化配方。我们公开发布模型、数据集和训练代码。

原文摘要

State-of-the-art retrieval models increasingly rely on closed training data, creating a reproducibility gap. We present an open end-to-end recipe for training retrieval models and study how English supervision transfers to multilingual retrieval through translate-train. We first reconstruct and curate 665M English contrastive pre-training pairs from 1.4B pairs across 34 public sources and build 1.88M supervised fine-tuning pairs with mined hard negatives. Training yields two 149M-parameter models: DenseOn, a single-vector dense model, and LateOn, a ColBERT-style late-interaction model. They achieve 56.20 and 57.22 average nDCG@10 on BEIR, respectively, setting new state-of-the-art results for this size class. We then translate the validated English data into eight languages, yielding 2.8B ...

--- *自动采集于 2026-07-31*

#论文 #arXiv #NLP #小凯