Language Model Re-rankers Are Fooled by Lexical Similarities (FEVER Workshop, ACL 2025)

Forum topic · 小凯 · 2026-07-05

Summary

This FEVER Workshop paper (ACL 2025) examines whether large language model (LLM) re-rankers genuinely improve retrieval in RAG pipelines. The authors evaluate six LLM re-ranking approaches across two question-answering datasets and find that, despite their sophistication, LLM re-rankers often fail to outperform traditional BM25 lexical retrieval, particularly on the Natural Questions dataset. Their analysis shows that LLM re-rankers are susceptible to lexical similarity traps: they tend to favor passages that superficially resemble the query—sharing entities or wording—even when those passages are irrelevant to the answer, while down-ranking dissimilar-looking but actually useful documents. The paper demonstrates that standard IR metrics may overstate re-ranker benefits and discusses dataset-level differences in how well lexical signals predict relevance. The work offers practical guidance for RAG system builders, suggesting that BM25 remains a strong, cheap baseline and that hybrid approaches combining lexical and neural signals deserve attention. The PDF is available at https://aclanthology.org/2025.fever-1.2.pdf.

Language Model Re-rankers Are Fooled by Lexical Similarities (FEVER @ ACL 2025)

Source PDF: https://aclanthology.org/2025.fever-1.2.pdf
Venue: Fact Extraction and VERification (FEVER) Workshop, ACL 2025
Track: Ranking for Search

Key points

Question studied: Do LLM-based re-rankers actually improve first-stage retrieval quality in retrieval-augmented generation (RAG) pipelines, or do they replicate lexical heuristics?
Findings: Across multiple datasets and several LLM re-ranking methods, the authors find LLM re-rankers often struggle to beat BM25, and are fooled by surface-level lexical similarity between query and passage.
Failure mode: Passages that look lexically similar (shared entities, wording) to the query can be ranked highly even when they do not answer it, while genuinely relevant but lexically dissimilar passages get down-ranked.
Implication for RAG: Improving retrieval-stage components may yield larger end-to-end gains than adding expensive LLM re-rankers; simple lexical signals remain a strong and cheap baseline.
Note: For exact quantitative results per dataset and per re-ranker, consult the original PDF tables.

Relevance to practitioners

Treat BM25 / hybrid retrieval as a mandatory baseline before adopting LLM re-rankers.
Evaluate re-rankers on metrics beyond aggregate nDCG to expose lexical-similarity biases.
Consider filtering or de-duplication to reduce the influence of near-lexical distractors in retrieved candidate sets.

Limitations

Results depend on dataset characteristics (e.g., Natural Questions vs. other QA corpora); conclusions may not transfer uniformly across domains.
LLM re-ranker performance evolves rapidly with newer models; reported gaps should be re-validated on current systems.

Related entries

A Thorough Comparison of Cross-Encoders and LLMs for Reranking SPLADE
Accelerating Listwise Reranking: Reproducing and Enhancing FIRST (SIGIR)
Adaptive Neural Ranking Framework: Toward Maximized Business Goal
Deep Learning to Rank in Industrial Search Engines