DISKCO: Disentangling Knowledge from Cross-Encoder to Bi-Encoder (WWW 2024)
Overview
DISKCO is a research paper published at WWW 2024, listed on Amazon Science's publication page. It targets a long-standing tension in neural information retrieval between the cross-encoder and bi-encoder architectures.
Problem: Cross-Encoders Are Accurate but Slow
- Cross-encoders concatenate the query and candidate document and pass them through a transformer together. This deep interaction yields state-of-the-art ranking accuracy, but document representations cannot be precomputed — every query-document pair requires a fresh forward pass, which is infeasible for first-stage retrieval over millions of documents.
- Bi-encoders encode queries and documents independently into dense vectors, enabling efficient approximate nearest-neighbor (ANN) search. However, the lack of token-level query-document interaction usually costs accuracy, especially on fine-grained relevance distinctions.
- Training cost vs. serving cost: distillation adds offline training expense but keeps online inference cheap — a favorable trade for high-QPS systems.
- Index updates: as with any bi-encoder, document embeddings must be re-encoded and re-indexed when the encoder or corpus changes; versioned embedding indexes and rollback plans are recommended.
- Evaluation: relevance gains should be validated both offline (nDCG@10, Recall@k, MRR on benchmarks) and online via interleaving or A/B experiments, since offline ranking metrics do not always translate to user-visible improvements.
- Deep Learning to Rank in Industrial Search Engines, Recommender Systems
- A Thorough Comparison of Cross-Encoders and LLMs for Reranking SPLADE
- Adaptive Neural Ranking Framework: Toward Maximized Business Goal
- Original publication: DISKCO: Disentangling knowledge from cross-encoder to bi-encoder — Amazon Science, WWW 2024.
Approach: Disentangling and Distilling Knowledge
DISKCO transfers the matching knowledge of a cross-encoder into a bi-encoder via knowledge distillation. The key idea is to disentangle the query-document knowledge contained in the cross-encoder's joint representation so that it can be effectively absorbed by a student bi-encoder, rather than naively matching final scores. The result is a retriever that keeps the inference-time efficiency of independent query/document encoding while closing part of the accuracy gap toward the cross-encoder teacher.
Why It Matters for Industrial Search
Modern large-scale search stacks typically use a cascade: a fast bi-encoder retriever (recall stage) followed by an expensive cross-encoder reranker (precision stage). By strengthening the bi-encoder with cross-encoder knowledge, DISKCO can:
1. Improve first-stage recall quality without added inference latency; 2. Potentially reduce the reranking depth needed downstream, cutting cost and tail latency; 3. Fit into standard dense-retrieval serving infrastructure (ANN indexes, cached document embeddings).