Sampling-Bias-Corrected Neural Modeling for Large Corpus Item Recommendations (Google, RecSys 2019)
Source: Google Research publication page
Overview
Modern large-corpus recommendation systems commonly use a two-tower (dual-encoder) neural model: one tower embeds the user/context and the other embeds candidate items into a shared vector space, where retrieval is performed with approximate nearest-neighbor search. A popular training strategy is the in-batch softmax: for each clicked (user, item) pair in a batch, all other items in the batch serve as negatives. This makes training highly efficient, since item embeddings can be shared across examples in the batch.
Problem: Sampling Bias
The in-batch softmax implicitly assumes that items are sampled uniformly at random as negatives. In reality, items appearing in training batches come from system logs (e.g., clicks), so sampling frequency is proportional to item popularity. This popularity-proportional sampling causes the model to systematically under-estimate scores for popular items, degrading retrieval quality. The bias term can be corrected by subtracting an estimate of the log sampling probability of each item.
Proposed Solution
- Derive a corrected in-batch softmax objective in which the logit of an item is adjusted by its estimated log sampling probability. The corrected objective is shown to be an upper bound of the (unbiased) log-likelihood of the true data distribution.
- Introduce a streaming frequency estimation algorithm that estimates the sampling probability of items on the fly with estimated count statistics, suitable for large-scale distributed training where maintaining exact per-item counts over hundreds of millions of items is impractical.
- The correction integrates cleanly into existing two-tower training pipelines with negligible overhead.
- Retrieval recall (ability of the candidate generator to surface relevant items), and
- Ranking quality metrics on downstream tasks.
- 360Brew: A Decoder-only Foundation Model for Personalized Ranking
- Actions Speak Louder than Words: Trillion-Parameter Sequential Transducers
- Augmenting Netflix Search with In-Session Adapted Recommendations (RecSys)
- Bridging Language and Items for Retrieval and Recommendation
- DiffKG: Knowledge Graph Diffusion Model for Recommendation (WSDM 2024)
Evaluation
Experiments cover retrieval and ranking tasks, including data from YouTube recommendations. Compared to in-batch softmax without correction, the proposed sampling-bias-corrected training improves:
Ablations analyze the effect of the streaming frequency estimator against oracle counts.
Key Takeaways
1. LogQ correction: subtracting the estimated log sampling probability of in-batch negatives removes popularity bias — this technique has become standard practice in industrial dual-encoder retrieval systems. 2. Streaming statistics make it practical: per-item counts can be estimated cheaply and in a distributed fashion, avoiding centralized frequency tables over huge corpora. 3. The method benefits both the retrieval stage (candidate generation) and the ranking stage when both are trained with sampled negatives.
Relevance
This paper is a foundational reference for large-scale neural retrieval. It is commonly cited alongside two-tower/dual-encoder work and is relevant to anyone building contrastively trained retrieval models where negatives come from logged traffic rather than uniform sampling.