Paper Overview
Research areas: cs.AI, cs.IR Authors: Koutian Wu, Junjie Zhou, Ergan Shang, Jiayu Wang, Pengqian Han, Junkai Wang, Wanghan Xu arXiv: 2609.11115
Abstract
Benchmark researchers and developers of large language models (LLMs) and other AI systems need to find relevant evaluations, locate their benchmark datasets and code, and understand the settings behind reported scores. We present Benchmark Radar, a living database and search engine for retrieval and discovery of AI benchmarks, covering LLM evaluation, agentic and tool-use benchmarks, coding, reasoning, safety, and domain-specific evaluations.
The system combines daily discovery of benchmark papers, repositories, datasets, and releases with a searchable benchmark catalog, mentions in model cards and technical reports, and score histories. It retains source identities and citations so readers can inspect candidate benchmarks and their evaluation evidence.
Key Details
- Daily discovery sources: 37 total — 13 direct connectors and 24 first-party research and engineering feeds
- Catalog size: 1,283 source records drawn from 4 benchmark catalogs
- Numeric observations: 12,916 observations across 790 records
- A web dashboard with a benchmark leaderboard
- A Pareto frontier view of score against measured use
- Saturation and trend views
- Daily feeds and downloadable evidence
- A command-line interface (CLI) for offline queries
- Reproducible analysis
Analysis and Release
The paper describes the collection and retrieval pipeline, audits the full catalog, and examines benchmark saturation, adoption trends, and the limits of score comparisons. A worked example walks through a complete prior-art search, showing how to query the catalog and inspect benchmark evidence when designing a new evaluation.
The release includes:
*Auto-collected on 2026-09-13*