Generating Query Recommendations via LLMs: GQR and RA-GQR
Paper: Generating Query Recommendations via LLMs Authors: Andrea Bacciu, Enrico Palumbo, Andreas Damianou, Nicola Tonellotto, Fabrizio Silvestri Published: 2024-05-30 | Source: https://arxiv.org/abs/2405.19749 | Category: Search Assistance
TL;DR
The paper reframes query recommendation as a generative task using LLMs. The proposed GQR requires no training or fine-tuning, works in cold-start scenarios, and its query-log-augmented variant RA-GQR achieves state-of-the-art NDCG@10 on Robust04 and ClueWeb09B.
Background & Motivation
Query recommendation systems are ubiquitous in modern search engines, helping users formulate effective queries. However, conventional systems need large amounts of data — indexed documents and query logs. Query logs are expensive to collect and maintain, unavailable in cold-start scenarios, and require complex, time-consuming cascading pipelines for creating, combining, and ranking recommendations.
Approach
1. GQR (Generative Query Recommendation): Uses an off-the-shelf LLM with a carefully designed prompt that teaches the model the recommendation task, even with a single example. No training or fine-tuning required. 2. RA-GQR (Retriever-Augmented GQR): Dynamically composes the prompt by retrieving similar queries from available query logs, improving quality when logs exist.
Both reuse pre-existing neural architectures, making the approach simpler and more ready-to-market — particularly valuable in cold-start settings.
Results
- GQR achieves state-of-the-art NDCG@10 and clarity score against two commercial search engines and the previous state of the art on Robust04 and ClueWeb09B, improving NDCG@10 by up to ~4% on average over the best competitor.
- RA-GQR further improves NDCG@10 by ~11% (Robust04) and ~6% (ClueWeb09B) over the best competitor.
- In a blind user study, the system won ~59% of user preferences, indicating the most engaging recommended queries.
- Asking Clarification Questions to Handle Ambiguity in Open-Domain QA (arXiv:2305.13808)
- Asking Clarifying Questions in Open-Domain Information-Seeking Conversations (DOI: 10.1145/3331184.3331265)
- DiAL: Diversity-aware listwise ranking for query auto-complete (EMNLP)
- Enhancing Discoverability in Enterprise Conversational Systems (arXiv:2412.10933)
- Evaluating auto-complete ranking for diversity and relevance (ECIR 2024)
- Evaluation and Continual Improvement for an Enterprise AI Assistant (arXiv:2407.12003)
- Original paper: Generating Query Recommendations via LLMs — arXiv:2405.19749
Original Abstract (English)
> Query recommendation systems are ubiquitous in modern search engines, assisting users in producing effective queries to meet their information needs. However, these systems require a large amount of data to produce good recommendations, such as a large collection of documents to index and query logs. In particular, query logs and user data are not available in cold start scenarios. Query logs are expensive to collect and maintain and require complex and time-consuming cascading pipelines for creating, combining, and ranking recommendations. To address these issues, we frame the query recommendation problem as a generative task, proposing a novel approach called Generative Query Recommendation (GQR). GQR uses an LLM as its foundation and does not require to be trained or fine-tuned to tackle the query recommendation problem. We design a prompt that enables the LLM to understand the specific recommendation task, even using a single example. We then improved our system by proposing a version that exploits query logs called Retriever-Augmented GQR (RA-GQR). RA-GQR dynamically composes its prompt by retrieving similar queries from query logs. GQR approaches reuses a pre-existing neural architecture resulting in a simpler and more ready-to-market approach, even in a cold start scenario. Our proposed GQR obtains state-of-the-art performance in terms of NDCG@10 and clarity score against two commercial search engines and the previous state-of-the-art approach on the Robust04 and ClueWeb09B collections, improving on average the NDCG@10 performance up to ~4% on Robust04 and ClueWeb09B w.r.t the previous best competitor. RA-GQR further improve the NDCG@10 obtaining an increase of ~11%, ~6% on Robust04 and ClueWeb09B w.r.t the best competitor. Furthermore, our system obtained ~59% of user preferences in a blind user study, proving that our method produces the most engaging queries.
Context within Generative IR
Neural information retrieval has evolved from BM25 through BERT cross-encoders, bi-encoder dense retrieval, and late interaction, toward generative retrieval and LLM agents. Generative approaches reduce cascading errors but face index-update challenges; LLM-based methods add semantic priors and cold-start capability, though online inference cost and hallucination risks demand careful system design. RAG-style and agentic search extend external knowledge access into an iterative, plannable process, shifting evaluation from static nDCG toward task success and process-level metrics.
Engineering Checklist
| Concern | Recommendation | |---|---| | Latency | Cascade + early stopping, cache popular queries, async re-ranking | | Quality | Interleaving experiments, human audits, citation verification | | Safety | Source whitelisting, poisoning detection, output filtering | | Cost | Route to smaller models, distillation, hybrid sparse+dense retrieval |