[论文] BEAKER: An Expert-Curated Benchmark for Embodied Brains in Self-Drivin...

论文概要

研究领域: ML 作者: Fei Lin, Tengchao Zhang, Ziyang Gong, Xiaotong Yu, Bohan Zhang, Yifan Zhou, Qihao Yang, Ji Dai, Dong Li, Yining Jiang, Qinghua Ni, Jun Huang, Qiang Zhang, Yue Zhou, Zhihang Zhong, Yonglin Tian, Fei-Yue Wang, Xue Yang 发布时间: 2026-11-04 arXiv: 2611.06911

中文摘要

自动驾驶化学实验室（SDCLs）正将化学实验推向具身自动化——机器必须感知实验室场景、推理实验状态，并在物理、程序和安全约束下行动。尽管多模态大语言模型（MLLMs）日益被视为此类系统的具身大脑，现有实验室基准主要关注安全性、异常检测或一般科学理解，对其面向执行的具身能力探索不足。为填补这一空白，我们提出 BEAKER——评估 SDCL 中具身可操作性、知识、实验和推理的基准。BEAKER 包含 1,000 个专家策划的问答样本，覆盖 11 种图像和视频任务类型，涵盖化学实验执行所需的关键判断：可供性定位、轨迹规划、设备/状态/安全识别、装置搭建、操作与步骤推理及时序过程推理。所有问题和标注由化学与具身 AI 专家手动创建和审核，不使用 AI 辅助生成。我们在离线单轮设置下评估 28 个通用、科学导向、化学专用和具身 MLLM。结果显示：当前模型能跟随视频中的粗粒度实验流程，但在将动作定位到可操作区域和可行轨迹、识别细粒度实验室设备/状态/安全规范、以及排列中等长度流程步骤方面仍有困难。BEAKER 揭示了粗粒度过程理解与实验室执行所需的精确、动作锚定感知之间的明显差距，为未来 SDCL 中的 MLLM 提供诊断基准。

原文摘要

Self-Driving Chemical Laboratories (SDCLs) are moving chemical experimentation toward embodied automation, where machines must perceive laboratory scenes, reason about experimental states, and act under physical, procedural, and safety constraints. Although Multimodal Large Language Models (MLLMs) are increasingly considered as embodied brains for such systems, existing laboratory benchmarks mainly focus on safety, anomaly detection, or general scientific understanding, leaving their execution-oriented embodied capabilities underexplored. To fill this gap, we introduce BEAKER, a Benchmark for evaluating Embodied Actionability, Knowledge, Experimentation, and Reasoning in SDCLs. BEAKER contains 1,000 expert-curated question-answer samples across 11 image and video task types, covering key j...

#论文 #arXiv #ML #小凯