Bradley-Terry Rankings for Recommender Systems Across Dataset Taxonomies
Paper Details
- Field: Machine Learning
- Authors: Ekaterina Grishina, Stepan Kuznetsov, Askar Tsyganov
- Published: 2025-06-11
- arXiv: 2506.08633
- A Bradley-Terry model-based methodology for fair, data-driven ranking of recommendation algorithms
- Evidence that rankings depend on key dataset statistics (sparsity, sequential structure, scale)
- A new metric for evaluating ranking consistency
- Robustness of the rankings to incomplete data
- A dataset-specific approach that predicts algorithm rankings on unseen datasets using BT trees and BT models with covariates
Abstract
The ranking of recommendation algorithms is a challenging problem since model performance is sensitive to dataset characteristics such as sparsity, sequential structure, and scale. This drives a demand for a proper methodology for fair comparison between algorithms. Naive aggregation of performance metrics (e.g., averaging NDCG over benchmarks) can yield misleading rankings, undermining practical selection.
To address this problem, the authors introduce a novel, data-driven ranking methodology based on the Bradley-Terry (BT) model. They demonstrate that the obtained ranking depends on key dataset statistics. Additionally, they propose a novel metric for evaluating ranking consistency and demonstrate the robustness of their ranking to incomplete data.
Finally, the paper introduces a dataset-specific methodology for ranking algorithms on unseen datasets without running the models, relying on extensions of the BT framework, including BT trees and BT models with covariates.