Beyond the Limitation of a Single Query: Training LLMs for Query Expansion with Reinforcement Learning (NVIDIA, Oct 2025)

Forum topic · 小凯 · 2026-07-05

Summary

This forum post summarizes an NVIDIA research paper (arXiv:2510.10009, October 2025) by Shu Zhao, Tan Yu, and Anbang Xu, which addresses the limitation of relying on a single user query in information retrieval. The work trains large language models to perform query expansion using reinforcement learning, aiming to improve retrieval quality by letting the model learn what expanded queries actually help downstream retrieval, rather than relying on heuristic or purely generative expansion. The post situates the paper within the broader search/recommendation landscape: the evolution from BM25 and dense retrieval to LLM-based and agentic search, where the number of retrieval steps and expansion strategies themselves become learnable decisions. It also discusses typical evaluation protocols (nDCG@10, MRR, Recall@k on benchmarks such as MS MARCO, BEIR, and Natural Questions), engineering considerations like latency budgets, caching, safety filtering, and cost control, and links to related work on aligned query expansion, query decomposition, and conversational query rewriting. Readers should verify quantitative results against the original PDF.

Beyond the Limitation of a Single Query: Train Your LLM for Query Expansion with Reinforcement Learning

This forum post introduces an NVIDIA research paper from October 2025, available on arXiv: https://arxiv.org/abs/2510.10009. Authors: Shu Zhao, Tan Yu, Anbang Xu.

Overview

The paper tackles a long-standing limitation in information retrieval: a user's original query is often insufficient to surface the best documents. Instead of treating the submitted query as fixed, the authors train an LLM to perform query expansion via reinforcement learning, so that the model learns to generate expanded queries based on whether they actually improve downstream retrieval performance, rather than relying on heuristic rules or purely generative expansion that may introduce noise or knowledge leakage.

Context within Search / Rec Systems

The post places this work in the evolution of neural IR:

From BM25 → dense bi-encoder retrieval → cross-encoder reranking → late interaction → generative retrieval → LLM-driven agentic search.
In the LLM era, new variables emerge: the inference budget and the action space (whether to retrieve, how many times, which tools to call). Query expansion becomes a learnable policy decision rather than a fixed pipeline step.

Typical Evaluation Setup

Papers in this area are commonly evaluated with:

Datasets: MS MARCO, BEIR, Natural Questions, domain-specific corpora
Metrics: nDCG@10, MRR, Recall@k, Hit@k, plus latency and token cost
Baselines: BM25, dense retrieval, cross-encoder reranking, no-retrieval LLM, commercial search APIs

Note: specific numerical results should be verified against the original paper PDF.

Engineering Takeaways

| Concern | Suggestion | |---------|-----------| | Latency | Cascaded retrieval with early stopping, caching popular queries, async reranking | | Quality | Interleaving experiments, human audits, citation verification to confirm offline gains transfer online | | Safety | Source whitelisting, poisoning/bias detection, output filtering for open retrieval | | Cost | Route to smaller models, distillation, hybrid sparse+dense retrieval |

Related Entries

Aligned Query Expansion: Efficient Query Expansion for IR (arXiv:2507.11042)
Decomposing Complex Queries for Tip-of-the-tongue Retrieval (arXiv:2305.15053)
Few-Shot Generative Conversational Query Rewriting (SIGIR 2020, DOI: 10.1145/3397271.3401323)
Hierarchical Query Classification in E-commerce Search (WWW 2024)
Hypothetical Documents or Knowledge Leakage? Rethinking LLM-based Query Expansion (arXiv:2504.14175)

Glossary

IR — Information Retrieval
RAG — Retrieval-Augmented Generation
nDCG — Normalized Discounted Cumulative Gain, a ranking quality metric
Agentic Search — Modeling search as sequential decision-making and tool invocation

Who Should Read This

1. Researchers: replicate core comparisons; check whether statistical significance and compute costs are reported. 2. Engineers: evaluate pluggable modules (retriever, reranker, planner) and integration cost with existing stacks. 3. Product managers: focus on user-perceivable benefits (latency, answer trustworthiness, multi-turn consistency) rather than offline nDCG alone.