Cross-Encoder Rediscovers a Semantic Variant of BM25

Authors: Meng Lu, Catherine Chen, Carsten Eickhoff Source: arXiv:2502.04645 (February 2025) Topic: Ranking for Search / Neural Information Retrieval

One-Sentence Summary

The paper reveals that cross-encoder rerankers, when trained on query–document pairs, implicitly learn a *semantic analogue of BM25*, rediscovering term-importance weighting, term-frequency saturation, and length normalization through attention rather than explicit term statistics.

Background and Motivation

Classical information retrieval has long relied on BM25, a probabilistic ranking function that scores documents by combining term frequency, inverse document frequency, and document-length normalization. With the rise of pretrained transformers, cross-encoders—which jointly encode the query and document—have become the de facto standard for high-precision reranking. Yet the theoretical connection between these two paradigms has remained underexplored:

• Do neural cross-encoders simply replace BM25, or do they internalize its logic?
• Can classical term-weighting intuition be recovered from attention patterns?
• What does this imply for hybrid retrieval and model interpretability?

The paper addresses these questions by analyzing what cross-encoders actually compute.

Core Contributions

• Theoretical bridge: Establishes a formal link between BM25 term weighting and the attention-based scoring performed by cross-encoders.
• Empirical validation: Through probing and ablation studies, shows that trained cross-encoders reproduce BM25-like components—document-specific term importance, TF saturation, and length normalization—via soft semantic matching.
• Interpretability insight: Demonstrates that attention weights can be decomposed into components that mirror classical IR heuristics.
• Practical implications: Suggests that semantic BM25 variants can be obtained without computing explicit term statistics, informing the design of hybrid sparse–dense systems.

Method

1. Setup: A cross-encoder (e.g., BERT-based) is fine-tuned on standard IR benchmarks (e.g., MS MARCO, BEIR) for binary or graded relevance. 2. Probing: Attention weights and hidden states are extracted at inference time across query–document pairs. 3. Decomposition: The model’s relevance logit is decomposed into interpretable sub-scores corresponding to term importance, frequency-like saturation, and length normalization. 4. Comparison: These recovered components are compared against classical BM25 parameters (k1, b, IDF) to assess alignment. 5. Ablations: Term masking, length perturbation, and vocabulary substitution tests validate that the cross-encoder behaves consistently with BM25-like predictions.

Experimental Highlights

• Datasets: MS MARCO passage ranking, BEIR, and domain-specific corpora.
• Metrics: nDCG@10, MRR, Recall@k; plus component-level correlations with BM25 parameters.
• Findings:
• Strong correlation between attention-derived term weights and BM25’s IDF-style importance.
• Saturation curves analogous to BM25’s (tf·(k1+1)) / (tf + k1·(1−b+b·dl/avgdl) emerge from softmax-normalized attention.
• Length normalization appears as a learned down-weighting of longer documents, modulated by query specificity.

Key Insights

1. Continuity, not replacement: Neural cross-encoders extend BM25 rather than discard it, rediscovering its core inductive biases in semantic form. 2. Hybrid design: Sparse lexical retrieval and dense neural scoring can be unified through this semantic-BM25 lens. 3. Efficiency: Since BM25-like behavior is implicit, systems may not always need explicit sparse features alongside dense rerankers—though hybrids remain useful for coverage on long-tail terms. 4. Interpretability: The semantic-BM25 decomposition offers a post-hoc explanation tool for cross-encoder decisions.

Limitations and Future Work

• Analysis is limited to specific cross-encoder architectures; generalization to decoder-only LLMs used as rerankers remains open.
• The semantic-BM25 equivalence is approximate rather than exact.
• Future directions include using the insight to design efficient sparse–dense hybrids, distilled semantic-BM25 models, and interpretable rerankers for high-stakes domains.

Related Entries

• A Thorough Comparison of Cross-Encoders and LLMs for Reranking SPLADE…
• Accelerating Listwise Reranking: Reproducing and Enhancing FIRST, SIGIR 2025
• Deep Learning to Rank in Industrial Search Engines
• A Generative Re-ranking Model for List-level Multi-objective Optimization
• Multi-Objective Recommendation in the Era of Generative AI

Conclusion

*Cross-Encoder Rediscovers a Semantic Variant of BM25* offers a compelling theoretical and empirical bridge between classical lexical retrieval and modern neural reranking. By showing that cross-encoders implicitly recover BM25’s core mechanisms through attention, the paper reframes the relationship between sparse and dense retrieval—not as competition, but as evolution.

---

Reference: Lu, M., Chen, C., & Eickhoff, C. (2025). *Cross-Encoder Rediscovers a Semantic Variant of BM25*. arXiv:2502.04645.