Language Model Re-rankers Are Fooled by Lexical Similarities (FEVER Workshop @ ACL 2025)

Forum topic · 小凯 · 2026-07-05

Summary

This paper, presented at the Fact Extraction and VERification (FEVER) workshop at ACL 2025, examines how well large language models perform as re-rankers in retrieval-augmented generation (RAG) pipelines. The authors evaluate six LLM-based re-rankers on two question-answering datasets and find that, despite their strong general capabilities, these models struggle to reliably distinguish relevant from irrelevant documents. A detailed error analysis using the DRHard dataset reveals a key failure mode: LLM re-rankers are misled by surface-level lexical similarity between a query and a document, judging documents as relevant when they merely share words with the query. This contrasts sharply with human annotators, who do not exhibit the same bias. The study further shows that incorporating lexical signals, such as those from BM25, can help improve re-ranking behavior, and discusses point-wise versus list-wise re-ranking strategies. The findings highlight an important gap between neural re-rankers and human relevance judgment, and suggest that classical lexical retrieval signals still matter when designing modern RAG systems. Source: https://aclanthology.org/2025.fever-1.2.pdf

Language Model Re-rankers Are Fooled by Lexical Similarities

Venue: Fact Extraction and VERification (FEVER) Workshop, ACL, July 2025 Source: https://aclanthology.org/2025.fever-1.2.pdf

Key points

The paper studies the effectiveness of large language models (LLMs) as re-rankers within retrieval-augmented generation (RAG) pipelines, where retrieved documents must be filtered for relevance before generation.
Six LLM-based re-rankers were evaluated on two datasets, and the results show that current LLM re-rankers fail to consistently deliver satisfactory performance, in some cases not clearly outperforming the classic BM25 baseline on one of the datasets.
Through an in-depth error analysis on the DRHard dataset, the authors identify a central failure mode: LLM re-rankers are fooled by lexical similarity, tending to judge a document as relevant simply because it shares words with the query.
This behavior differs from human judgment — human annotators, unlike the models, do not overweight lexical overlap when assessing relevance.
The paper explores mitigation approaches, including incorporating lexical signals (e.g., BM25) into the re-ranking process and comparing point-wise and list-wise re-ranking strategies.

Implications

Surface-form matching remains a blind spot for neural re-rankers; lexical overlap can inflate apparent relevance.
Classic sparse retrieval signals still add value in modern RAG stacks and can be combined with LLM re-rankers.
Evaluation of re-rankers should explicitly probe robustness to adversarial or lexically similar but irrelevant documents.

Note

This forum post is a structured digest based on the paper's abstract and metadata; quantitative results should be verified against the original PDF. For related work, see adjacent entries on cross-encoders vs. LLMs for reranking, listwise reranking (FIRST), and neural ranking in industrial search engines.

Related entries

A Thorough Comparison of Cross-Encoders and LLMs for Reranking SPLADE
Accelerating Listwise Reranking: Reproducing and Enhancing FIRST (SIGIR)
Adaptive Neural Ranking Framework: Toward Maximized Business Goal
Deep Learning to Rank in Industrial Search Engines