[论文] [论文] Auto-Research Arena: Benchmarking and Advancing Auto-Research in ...

论文概要

研究领域: ML
作者: Xi Chen, Chongjian Ge, Yiran Xu, Yiqun Mei, Henrik I. Christensen et al.
发布时间: 2026-10-26
arXiv: 2610.35866

中文摘要

随着自主研究系统能力不断增强，基准测试既要评估其表现，也要支撑有意义的研究进展。我们提出 Auto-Research Arena——评估自主研究系统并在语言建模领域发现新想法的多目标基准。Arena 在共享实验环境中支持多样化研究目标，核心设计是把可采纳性与改进分离：模型质量与资源约束决定结果是否合格，所选目标衡量合格结果之间的进步。我们在语言模型预训练上实例化 Arena，设九个涵盖模型、数据、训练与推理效率的目标，用两个基础模型评估七个研究框架。Arena 清晰区分了现有框架在不同目标上的表现——若干复杂方法甚至输给简单的顺序搜索基线，说明当前自动研究系统仍有巨大提升空间。除评估外，Arena 还揭示出与活跃研究方向高度契合的、更广阔的模型与系统创意空间。把这些发现复用于新的质量驱动搜索后，同等数据与 FLOPs 下训练损失更低，同模型规模下 held-out BPB 更低。

原文摘要

As autonomous research systems become more capable, benchmarks must both evaluate their performance and support meaningful research progress. We introduce Auto-Research Arena, a multi-target benchmark for evaluating autonomous research systems and discovering ideas in language modeling. Arena supports diverse research objectives within a shared experimental environment by separating admissibility from improvement: model quality and resource constraints determine whether a result qualifies, while the selected target measures progress among admissible results. We instantiate Arena in language model pretraining with nine targets spanning model, data, training, and inference efficiency, and evaluate seven research harnesses using two foundation models. Arena clearly differentiates existing harnesses across research objectives, with several sophisticated methods outperformed by our simple sequential search baseline, indicating substantial headroom in current auto-research systems. Beyond evaluation, Arena reveals a substantially broader frontier of model and system ideas aligned with active research directions. Reusing these discoveries in a new quality-driven search yields lower training loss under matched data and FLOPs and lower held-out BPB at the same model size. Together, Arena provides a common framework for benchmarking autonomous research and expanding its scope in language modeling. The code is available at https://github.com/Auto-Research-Arena/auto-research-arena

自动采集于 2026-10-28

#论文 #arXiv #ML #小凯