Hypothetical Documents or Knowledge Leakage? Rethinking LLM-based Query Expansion (arXiv, Apr 2025)
This entry summarizes the arXiv paper "Hypothetical Documents or Knowledge Leakage? Rethinking LLM-based Query Expansion" (arXiv:2504.14175), authored by Yejun Yoon, Jaeyoon Jung, Seunghyun Yoon, and Kunwoo Park, filed under the Query Understanding section of the source list.
Key points
- Topic: LLM-based query expansion, most prominently the HyDE (Hypothetical Document Embeddings) family of methods, where an LLM generates a hypothetical answer document for a query, and retrieval is performed by embedding that generated document instead of the raw query.
- Central question: Do the observed retrieval improvements reflect genuine understanding of user intent, or do they result from knowledge leakage — i.e., the LLM's parametric memory already contains the benchmark's target documents, allowing it to effectively reconstruct the answer during expansion?
- Implication: If leakage is the dominant mechanism, reported benchmark gains overstate how well these methods will generalize to private corpora, fresh data, or domains unseen during the LLM's pretraining.
- Architecture: cascade retrieval + reranking + generation remains mainstream, but synthetic expansion data must be checked for leakage and distribution shift.
- Evaluation: offline metric gains should be cross-validated with human assessment and, ideally, fresh or non-public corpora to rule out memorization effects.
- Production constraints: latency, token cost, and safety filtering remain hard requirements; academic benchmarks alone do not justify deployment.
- Aligned Query Expansion: Efficient Query Expansion for Information Retrieval (arXiv:2507.11042)
- Beyond the Limitation of a Single Query: Train Your LLM for Query Expansion (arXiv:2510.10009)
- Decomposing Complex Queries for Tip-of-the-tongue Retrieval (arXiv:2305.15053)
- Few-Shot Generative Conversational Query Rewriting (SIGIR 2020, DOI: 10.1145/3397271.3401323)
- LLM-Based Query Expansion with Gaussian Kernel Semantic Enhancement
Context within IR research
The paper sits at the intersection of neural information retrieval and LLM-era system design:
1. Representation: queries and documents encoded as dense/sparse vectors, or hypothetical documents produced by generative models; 2. Pipeline: retriever → reranker → generator, with agentic paradigms increasingly making the number and strategy of retrieval steps learnable; 3. Evaluation: standard setups use benchmarks such as MS MARCO and BEIR with metrics like nDCG@10, MRR, and Recall@k — precisely the settings most susceptible to knowledge leakage effects.
Takeaways for practitioners
Limitations and future work
Typical open questions in this line of research include evaluation credibility under memorization, latency and cost trade-offs, hallucination risks in generated pseudo-documents, and cross-lingual generalization. Readers should consult the original PDF for exact experimental numbers before citing quantitative results.