Domain-specific Question Answering with Hybrid Search (arXiv 2412.03736)
Overview
- Paper: Domain-specific Question Answering with Hybrid Search
- Authors: Dewang Sultania, Zhaoyu Lu, Twisha Naik, Franck Dernoncourt, David Seunghyun Yoon, Sanat Sharma, et al. (14 authors)
- Published: December 2024 (arXiv)
- Topic area: Hybrid search vs. vector search; domain-specific question answering
- The work targets question answering over specialized domains, where generic LLMs lack reliable, up-to-date knowledge and must rely on external retrieval.
- It uses hybrid search, combining sparse (e.g., BM25-style lexical) retrieval with dense embedding-based retrieval to improve both recall and precision over specialized corpora.
- Datasets/benchmarks: MS MARCO, BEIR, Natural Questions, and domain-specific corpora.
- Metrics: nDCG@10, MRR, Recall@k, Hit@k, task success rate, latency, and token cost.
- Baselines: BM25, dense retrieval, cross-encoder re-ranking, retrieval-free LLMs, commercial search APIs.
- IR — Information Retrieval
- RAG — Retrieval-Augmented Generation
- LTR — Learning to Rank
- nDCG — Normalized Discounted Cumulative Gain, a ranking-quality metric
- Agentic Search — modeling search as sequential decision-making with tool calls
- Gen-IR — Generative Information Retrieval
- Azure AI Search: Outperforming vector search with hybrid retrieval
- COS-Mix: Cosine Similarity and Distance Fusion (arXiv 2406.00638)
- Deep Retrieval at CheckThat! 2025 (arXiv 2505.23250)
- Efficient Knowledge Graph Construction and Retrieval (arXiv 2507.03226)
- Hybrid Hierarchical Retrieval for Open-Domain QA
- Modernizing Facebook Scoped Search (arXiv 2509.13603)
- Original paper: https://arxiv.org/abs/2412.03736
Key points
Context: why hybrid search matters
Classic search stacks follow a funnel: recall (retrieval), discrimination (ranking), and presentation (generation). Dense retrieval enables millisecond-level ANN recall but is sensitive to domain shift and long-tail queries; cross-encoder re-rankers are accurate but cannot precompute document representations; hybrid approaches mitigate these weaknesses by fusing lexical and semantic signals.
In the LLM era, RAG and agentic search extend external knowledge access from a one-shot retrieval step into an iterative, verifiable, plannable process. Evaluation correspondingly shifts from static nDCG toward task success rate, citation accuracy, and multi-hop reasoning completeness.
Typical pipeline structure
1. Input & representation: encode queries, documents, and user context as dense or sparse representations, or structured prompts. 2. Core modules: retriever, re-ranker, planner, memory, tool interfaces, chained or parallelized per task. 3. Learning strategies: supervised fine-tuning, contrastive learning, distillation, RL, or synthetic data bootstrapping. 4. Inference strategies: single-shot retrieval, iterative retrieval, parallel sub-queries, early stopping, and budget control.
Evaluation
Note: exact quantitative results should be verified against the paper's PDF.
Engineering checklist
| Check | Question | Suggestion | |-------|----------|------------| | Data | Does training/indexing data contain PII? | Partitioned indexes, anonymization, rollback-able embedding versions | | Latency | What is the p99 budget? How many retrieval steps? | Cascade + early stop, query caching, async re-ranking | | Quality | Do offline gains translate to online satisfaction? | Interleaving experiments, human audits, citation checks | | Safety | Does open retrieval introduce poisoning/bias? | Source allowlists, adversarial detection, output filtering | | Cost | Token and GPU cost per query? | Small-model routing, distillation, sparse+dense hybrid |