Benchmark Everything Everywhere All at Once: Benchmark Agent for Fully Autonomous Benchmark Construction

Forum topic · 小凯 · 2026-06-08

Summary

This paper introduces Benchmark Agent, a fully autonomous agentic system designed to automate the construction of benchmarks for large language models (LLMs) and multimodal LLMs (MLLLMs). Motivated by the fact that benchmark building is labor-intensive, hard to reuse, and often suffers from rapid performance saturation that reduces discrimination among state-of-the-art models, the framework orchestrates the complete benchmark construction pipeline—from user query analysis and subtask design to data annotation and quality control. The authors evaluate Benchmark Agent by implementing 15 representative benchmarks spanning text understanding, multimodal understanding, and domain-specific reasoning. Extensive experiments using human evaluation, LLM-as-a-judge, and consistency checks show that the system can generate high-quality benchmark samples with minimal human involvement. Continuous evaluation further reveals notable findings, including that current models still struggle with certain domain-specific reasoning tasks. The work is available on arXiv (2606.06462), with a demo page and code repository to be released publicly.

Overview

Field: Machine Learning Authors: Shiyun Xiong, Dongming Wu, Peiwen Sun, Yuang Ai, Bokang Yang, Wencheng Han, Xiao-Hui Li, Xiangyu Yue Published: 2026-06-04 arXiv: 2606.06462

English Translation of the Abstract

Benchmarks are fundamental for evaluating and advancing LLMs and multimodal LLMs by providing standardized and explicit measures of performance. However, their construction is labor-intensive and hard to reuse, raising concerns about sustainability and scalability. Moreover, existing benchmarks often quickly reach performance saturation after their release, resulting in insufficient discrimination among state-of-the-art models.

To address these challenges, the authors introduce Benchmark Agent, a fully autonomous agentic system designed for benchmark building. The framework orchestrates the complete benchmark construction pipeline, from user query analysis and subtask design to data annotation and quality control.

To assess Benchmark Agent, the authors implemented it to produce 15 representative benchmarks spanning diverse evaluation scenarios, including text understanding, multimodal understanding, and domain-specific reasoning. Extensive experiments—including human evaluation, LLM-as-a-judge evaluation, and consistency checks—demonstrate that Benchmark Agent can generate high-quality benchmark samples with minimal human involvement. More importantly, through continuous evaluation, they observed several insightful findings, including that current models struggle with certain domain-specific reasoning tasks.

The team believes that rapidly evolving benchmarks can make important contributions to the research community. A preview and code will be made publicly available on a demo page and code repository.

Key Highlights

Problem: Benchmark construction is labor-intensive, hard to reuse, and prone to rapid performance saturation.
Solution: A fully autonomous agent system covering the full benchmark pipeline: query analysis, subtask design, data annotation, and quality control.
Evaluation: 15 representative benchmarks implemented across text understanding, multimodal understanding, and domain-specific reasoning.
Findings: High-quality samples with minimal human effort; current models still show weaknesses on some domain-specific reasoning tasks.

--- *Auto-collected on 2026-06-08.*