CAME: Competitively Learning a Mixture-of-Experts Model for First-Stage Retrieval
Metadata
| Field | Content | |-------|---------| | Title | CAME: Competitively Learning a Mixture-of-Experts Model for First-Stage Retrieval | | Published | January 2025 | | Venue | ACM | | Source link | https://dl.acm.org/doi/abs/10.1145/3678880 | | Type | Academic paper | | Section | Retrieval |
One-line Summary
CAME proposes a Mixture-of-Experts model trained via competitive learning to improve first-stage retrieval under strict efficiency constraints.
Background and Motivation
In large-scale search, recommendation, and personalization systems, information retrieval faces persistent challenges in efficiency, scalability, and user-intent understanding. Traditional pipeline approaches treat candidate generation, ranking, and generation separately, which limits adaptability in the LLM era where users expect natural-language interaction, multi-hop reasoning, and real-time knowledge.
First-stage retrieval is particularly sensitive to the efficiency–effectiveness trade-off: it must process millions of documents within milliseconds. The CAME paper addresses this by learning a Mixture-of-Experts architecture in which experts compete during training, encouraging specialization across query and document types while routing keeps per-query compute bounded.
Core Contributions
- A competitive learning scheme for MoE-based retrieval, where expert sub-networks specialize in different query/document patterns.
- An architecture designed to fit first-stage retrieval latency budgets while improving over dense bi-encoder baselines.
- Positioning within the broader evolution of neural IR: from BM25 → dense retrieval → late interaction → generative retrieval and LLM-based agents.
- Sparse retrieval (BM25): fast and robust, but lexical-only.
- Dense bi-encoders: millisecond-level recall via approximate nearest-neighbor search, but sensitive to domain shift and long-tail queries.
- Cross-encoders: high accuracy but cannot precompute document representations.
- Generative retrieval and LLM agents: reduce cascade errors but face index-update and cost challenges.
- CoEvo: Coevolution of LLM and Retrieval Model for Domain-Specific Information Retrieval
- DRAMA: Diverse Augmentation from Large Language Models to Smaller Dense Retrievers
- ExpandR: Teaching Dense Retrievers Beyond Queries with LLM Guidance
- Fine-Tuning LLaMA for Multi-Stage Text Retrieval
- How Does Generative Retrieval Scale to Millions of Passages?
- Original paper: https://dl.acm.org/doi/abs/10.1145/3678880
Context in Neural Information Retrieval
Neural IR has evolved through several generations, each balancing effectiveness, efficiency, and maintainability:
CAME belongs to the line of work exploring MoE architectures as a way to add model capacity without proportional inference cost.
Experimental Considerations
Papers in this area are typically evaluated on datasets such as MS MARCO, BEIR, and Natural Questions, using metrics like Recall@k, MRR, and nDCG@10, against baselines including BM25 and dense retrieval. Specific quantitative results for CAME should be verified against the original PDF; this entry is based on the abstract and public metadata.
Takeaways for Practitioners
1. Architecture: cascade retrieval + reranking + generation remains mainstream, but capacity-efficient designs (MoE, distillation, sparse-dense hybrids) are key for the recall stage. 2. Latency: p99 budgets dictate how much model capacity can be routed per query; caching and early stopping remain essential. 3. Evaluation: offline metric gains must be validated against online satisfaction; LLM-as-judge results should be cross-checked with human evaluation.