INTERS: Unlocking the Power of Large Language Models in Search with Instruction Tuning (arXiv 2401.06532)
Metadata
| Field | Content | |-------|---------| | Title | INTERS: Unlocking the Power of Large Language Models in Search with Instruction Tuning | | Authors | Yutao Zhu, Peitian Zhang, Chenghao Zhang, Yifei Chen, Binyu Xie, Zheng Liu, et al. (8 authors total) | | Published | January 2024 | | Source | https://arxiv.org/abs/2401.06532 | | Type | Academic paper |
Background and Motivation
In large-scale search, recommendation, and personalization systems, information retrieval has long faced challenges around efficiency, scalability, and user intent understanding. Traditional pipeline-based approaches separate retrieval, ranking, and generation, making it hard to meet the modern demands of natural language interaction, multi-hop reasoning, and real-time knowledge in the era of large language models (LLMs). INTERS was proposed in this context to systematically advance the theory and practice at the intersection of instruction tuning and search, covering open-domain information access, enterprise knowledge retrieval, conversational search, semantic understanding in recommender systems, and end-to-end architectures that couple external knowledge sources with generative models.
Core Contributions
- Proposes a unified view for the problem domain, organizing scattered related work into a comparable framework.
- Provides a clear decomposition of method components (representation learning, retriever, re-ranker, planner, generator, feedback mechanisms) to support engineering adoption.
- Offers reproducible benchmarks, datasets, or taxonomies in its experimental protocol, lowering the entry barrier for follow-up research.
- Discusses interfaces with emerging paradigms such as LLM tool calling, reinforcement learning, and multi-agent collaboration, and paths from research prototypes to industrial systems.
- Lists open problems: evaluation reliability, latency and cost, hallucination and safety, and cross-lingual / multimodal extension.
- Datasets: MS MARCO, BEIR, Natural Questions, domain-specific corpora, and public recommendation datasets.
- Metrics: nDCG@10, MRR, Recall@k, Hit@k, human preference, task success rate, latency, and token cost.
- Baselines: BM25, dense retrieval, cross-encoder re-ranking, retrieval-free LLMs, and commercial search APIs.
- Ablations: validating the contribution of each module (number of retrieval steps, re-ranking depth, training data scale) to final quality.
- A Comprehensive Study of Knowledge Editing for Large Language Models
- Recommendation as Instruction Following: A Large Language Model Empowered Recommender System
- RouteLLM: Learning to Route LLMs with Preference Data, Jun 2024, arXiv
- Translational Generative Retrieval via Potential Query Generation
- Zero-shot Document Retrieval with Hybrid Pseudo-document Retriever
- Representation Learning with Large Language Models for Recommendation
- Original paper: INTERS: Unlocking the Power of Large Language Models in Search with Instruction Tuning. arXiv: https://arxiv.org/abs/2401.06532
Method / System Architecture
The methodology follows a four-step pattern: problem formulation → model/system design → training or construction → inference pipeline.
1. Input and representation: encode queries, documents, and user context into dense or sparse representations, or construct structured prompts. 2. Core modules: may include a retriever, re-ranker, planner, memory module, and tool interfaces, connected in series or in parallel. 3. Learning strategies: supervised fine-tuning, contrastive learning, distillation, reinforcement learning (including process rewards), and bootstrapped data synthesis. 4. Inference strategies: single-pass retrieval, iterative retrieval, parallel sub-queries, early stopping, and budget control.
Experiments and Evaluation
Typical evaluation setups in this area include:
Specific numerical results should be verified against the original PDF; this report is based on the abstract and public metadata.
Key Takeaways
1. Architecture: cascaded retrieval + re-ranking + generation remains mainstream, but agentic paradigms are making "how many times to retrieve and with what strategy" itself a learnable object. 2. Data: high-quality instruction data and click/session logs are both critical; synthetic data requires safeguards against knowledge leakage and distribution shift. 3. Evaluation: the gap between offline metrics and online satisfaction is widening; LLM-as-judge needs cross-validation with human assessment. 4. Product: latency, cost, explainability, and safety policies are hard constraints for industrial deployment—academic benchmarks alone are insufficient.
Limitations and Future Work
Likely limitations include experiment scale constrained by GPU budget, mismatch between benchmarks and real user distributions, unknown cross-lingual generalization due to English-centric data, and safety risks of agent systems on the open web. Future directions include more efficient test-time compute allocation, deeper integration with knowledge graphs and structured databases, and causal/fairness constraints for recommender systems.
Glossary
| Term | Meaning | |------|---------| | IR | Information Retrieval | | RAG | Retrieval-Augmented Generation | | LTR | Learning to Rank | | nDCG | Normalized Discounted Cumulative Gain, a ranking quality metric | | Agentic Search | Modeling search as sequential decision-making and tool calling | | Gen-IR | Generative Information Retrieval |