NVIDIA Paper: Training LLMs for Query Expansion with Reinforcement Learning (arXiv 2510.10009)

Forum topic · 小凯 · 2026-07-05

Summary

An October 2025 NVIDIA research paper, 'Beyond the limitation of a single query: Train your LLM for query expansion with Reinforcement Learning' (arXiv:2510.10009), by Shu Zhao, Tan Yu, and Anbang Xu, addresses a core limitation of modern information retrieval: relying on a single user query for retrieval. The work trains a large language model to expand queries using reinforcement learning, aiming to improve retrieval quality in search and RAG systems. Rather than hand-crafted expansion rules or purely supervised methods, the RL approach optimizes the expansion behavior directly against downstream retrieval objectives. This forum post summarizes the paper's position in the query-understanding literature, its methodological framing (retriever, reranker, planner, and feedback components), typical evaluation protocols using benchmarks like MS MARCO and BEIR with metrics such as nDCG@10 and Recall@k, and engineering considerations for production deployment including latency, cost, safety, and evaluation reliability. Readers should consult the original PDF for exact quantitative results.

Beyond the Limitation of a Single Query: Train Your LLM for Query Expansion with Reinforcement Learning (NVIDIA, Oct 2025)

Source: arXiv:2510.10009 Authors: Shu Zhao, Tan Yu, Anbang Xu (NVIDIA) Category: Query Understanding / Information Retrieval

Overview

This October 2025 NVIDIA paper tackles a long-standing limitation in information retrieval: the dependence on a single, often ambiguous user query. The authors propose training an LLM to perform query expansion using reinforcement learning, so the model learns expansion strategies that directly improve downstream retrieval performance rather than merely mimicking supervised expansion examples.

Background

Traditional retrieval pipelines separate retrieval, ranking, and generation, which limits adaptability in the LLM era where users expect natural-language interaction, multi-hop reasoning, and up-to-date knowledge. Query expansion—rewriting or augmenting a raw query with additional terms or sub-queries—is a key lever for closing the vocabulary and intent gap between users and document collections. Prior approaches include pseudo-relevance feedback, aligned query expansion, and LLM-based hypothetical document generation; this work explores RL as a way to optimize expansion behavior against measurable retrieval outcomes.

Method framing

The paper fits into a standard modern pipeline:

1. Input & representation — encode the query, documents, and user context into dense/sparse representations or structured prompts. 2. Core modules — retriever, reranker, planner/expander, and feedback mechanisms. 3. Learning strategy — reinforcement learning applied to the query-expansion LLM, complementing SFT, distillation, and data synthesis. 4. Inference strategy — single-shot retrieval, iterative retrieval, parallel sub-queries, with early stopping and budget control.

> Note: Exact quantitative results and reward design details should be verified against the original PDF, as this post summarizes based on public metadata.

Evaluation context

Work in this area is typically evaluated on benchmarks such as MS MARCO, BEIR, and Natural Questions, using metrics like nDCG@10, MRR, and Recall@k, against baselines including BM25, dense retrieval, and cross-encoder rerankers.

Key takeaways for Search / RAG practitioners

Architecture: Cascade retrieve–rerank–generate remains dominant, but agentic paradigms increasingly treat *when and how many times to retrieve* as learnable decisions.
Data: High-quality instruction data and click/session logs matter; synthetic expansion data risks knowledge leakage and distribution shift.
Evaluation: Offline metric gains do not always translate to online satisfaction; LLM-as-judge should be cross-validated with human evaluation.
Production constraints: Latency, cost, explainability, and safety are hard requirements—don't optimize academic benchmarks alone.

Related entries

Aligned Query Expansion: Efficient Query Expansion for IR (arXiv:2507.11042)
Decomposing Complex Queries for Tip-of-the-tongue Retrieval (arXiv:2305.15053)
Few-Shot Generative Conversational Query Rewriting (SIGIR 2020)
Hypothetical Documents or Knowledge Leakage? Rethinking LLM-based Query Expansion (arXiv:2504.14175)

Glossary

| Term | Meaning | |------|---------| | IR | Information Retrieval | | RAG | Retrieval-Augmented Generation | | nDCG | Normalized Discounted Cumulative Gain, a ranking quality metric | | Agentic Search | Modeling search as sequential decision-making with tool calls | | Gen-IR | Generative Information Retrieval |