Evaluating Auto-Complete Ranking for Diversity and Relevance (ECIR 2025)

This post is a Chinese forum's annotated entry for an Amazon Science publication: "Evaluating auto-complete ranking for diversity and relevance" (ECIR 2025).

• Source: https://www.amazon.science/publications/evaluating-auto-complete-ranking-for-diversity-and-relevance
• Category: Search Assistance / industrial publication

Key points

• The work targets query auto-complete ranking, a core component of large-scale search, where suggestions must balance relevance to user intent with diversity of the candidate list.
• The entry frames the problem within the modern search stack: recall (coverage), re-ranking (discrimination), and generation/presentation, with LLM-era additions of inference budget and action space (whether and how often to retrieve, which tools to call).
• Typical system pipeline described:
1. Input & representation — encode queries, documents, and user context into dense/sparse representations or structured prompts; 2. Core modules — retrievers, re-rankers, planners, memory, and tool interfaces; 3. Learning strategies — supervised fine-tuning, contrastive learning, distillation, RL (including process rewards), and synthetic data bootstrapping; 4. Inference strategies — single-pass retrieval, iterative retrieval, parallel sub-queries, early stopping, and budget control.

Evaluation conventions discussed

• Datasets: MS MARCO, BEIR, Natural Questions, proprietary domain corpora, and public recommendation sets.
• Metrics: nDCG@10, MRR, Recall@k, Hit@k, human preference, task success rate, latency, and token cost.
• Baselines: BM25, dense retrieval, cross-encoder re-ranking, retrieval-free LLMs, and commercial search APIs.
• The post explicitly notes that exact numerical results should be verified against the original PDF, as the entry is based on the abstract and public metadata.

Engineering takeaways (from the post's checklist)

| Concern | Suggested practice | |---|---| | Data | PII partitioning, sanitization, versioned/rollback-safe embeddings | | Latency | Cascaded retrieval with early stopping, query caching, async re-ranking | | Quality | Interleaving experiments, human audits, citation verification | | Safety | Source whitelisting, poisoning/bias detection, output filtering | | Cost | Model routing, distillation, hybrid sparse+dense retrieval |

Related entries cross-referenced in the post

• Asking Clarification Questions to Handle Ambiguity in Open-Domain QA (arXiv:2305.13808)
• Asking Clarifying Questions in Open-Domain Information-Seeking Conversations (ACM, DOI 10.1145/3331184.3331265)
• DiAL: Diversity-aware listwise ranking for query auto-complete (EMNLP) — the most directly related prior work
• Enhancing Discoverability in Enterprise Conversational Systems (arXiv:2412.10933)
• Evaluation and Continual Improvement for an Enterprise AI Assistant (arXiv:2407.12003)
• Generating Query Recommendations via LLMs (arXiv:2405.19749)

Glossary (from the original post)

• IR — Information Retrieval
• RAG — Retrieval-Augmented Generation
• LTR — Learning to Rank
• nDCG — Normalized Discounted Cumulative Gain
• Agentic Search — modeling search as sequential decision-making with tool calls
• Gen-IR — Generative Information Retrieval

> Note: Much of the original post is boilerplate analysis appended to a short abstract. The paper's concrete contributions and numbers should be read from the Amazon Science page directly.