Hypothetical Documents or Knowledge Leakage? Rethinking LLM-Based Query Expansion

Forum topic · 小凯 · 2026-07-05

Summary

This forum post summarizes the arXiv paper 'Hypothetical Documents or Knowledge Leakage? Rethinking LLM-based Query Expansion' (arXiv:2504.14175) by Yejun Yoon, Jaeyoon Jung, Seunghyun Yoon, and Kunwoo Park. The paper critically examines LLM-based query expansion techniques such as HyDE (Hypothetical Document Embeddings), asking whether their measured retrieval gains genuinely come from expanding the query or from knowledge leakage—i.e., the LLM having memorized relevant documents during pretraining. The post places the work in the context of modern information retrieval pipelines (retriever, reranker, generator), lists related research on query expansion and conversational query rewriting, and provides practical engineering guidance on latency, cost, safety, and evaluation for search and recommendation systems. Readers are advised to consult the original PDF for exact quantitative results, since the post primarily reproduces metadata, background discussion, and cross-references within an awesome-list collection on query understanding.

Hypothetical Documents or Knowledge Leakage? Rethinking LLM-Based Query Expansion

Source: arXiv:2504.14175 — Yejun Yoon, Jaeyoon Jung, Seunghyun Yoon, Kunwoo Park (April 2025)

This forum post introduces and contextualizes the paper *Hypothetical Documents or Knowledge Leakage? Rethinking LLM-based Query Expansion*. The central question the paper raises: when LLM-generated hypothetical documents improve retrieval, is the gain due to genuine query expansion, or is it actually knowledge leakage — the model recalling documents it saw during pretraining?

Key points

Topic: LLM-based query expansion in information retrieval, critically re-examining methods like HyDE (Hypothetical Document Embeddings).
Core hypothesis: Reported improvements of hypothetical-document approaches may partly stem from the LLM's memorization of corpus content rather than from the expansion mechanism itself.
Context: The paper sits at the intersection of IR and LLM systems, where classic retrieve → rerank → generate pipelines are being rethought, and where inference budget and tool-use/action space become new design variables.

Background (as presented in the post)

The post frames the work within large-scale search, recommendation, and personalization systems. Traditional pipelines treat retrieval, ranking, and generation separately, while LLM-era systems demand natural-language interaction, multi-hop reasoning, and up-to-date knowledge. Relevant scenarios include open-domain search, enterprise knowledge retrieval, and conversational search.

Typical evaluation setup discussed

Datasets: MS MARCO, BEIR, Natural Questions, domain-specific corpora
Metrics: nDCG@10, MRR, Recall@k, Hit@k, latency, token cost
Baselines: BM25, dense retrieval, cross-encoder reranking, retrieval-free LLMs

Takeaways for practitioners

1. Architecture: cascade retrieval + rerank + generation remains mainstream, but agentic paradigms treat retrieval count and policy as learnable. 2. Data: high-quality instruction data and click/session logs matter; synthetic data must guard against leakage and distribution shift. 3. Evaluation: gaps between offline metrics and online satisfaction are widening; LLM-as-judge should be cross-validated with human assessment. 4. Deployment: latency, cost, interpretability, and safety are hard production constraints, not just benchmark numbers.

> Note: The post is largely a metadata-driven summary; exact quantitative results should be verified against the original PDF at https://arxiv.org/abs/2504.14175.

Hypothetical Documents or Knowledge Leakage? Rethinking LLM-Based Query Expansion

Summary

Hypothetical Documents or Knowledge Leakage? Rethinking LLM-Based Query Expansion

Key points

Background (as presented in the post)

Related entries cross-referenced in the post

Typical evaluation setup discussed

Takeaways for practitioners

Tags