Paper Overview
- Field: Machine Learning
- Authors: Fei Lin, Tengchao Zhang, Ziyang Gong, Xiaotong Yu, Bohan Zhang, Yifan Zhou, Qihao Yang, Ji Dai, Dong Li, Yining Jiang, Qinghua Ni, Jun Huang, Qiang Zhang, Yue Zhou, Zhihang Zhong, Yonglin Tian, Fei-Yue Wang, Xue Yang
- arXiv: 2611.06911
- BEAKER contains 1,000 expert-curated question-answer samples across 11 image and video task types.
- Tasks cover key judgments required for chemical experiment execution: affordance grounding, trajectory planning, equipment/state/safety recognition, setup construction, operation and step reasoning, and temporal process reasoning.
- All questions and annotations were manually created and reviewed by chemistry and embodied AI experts, with no AI-assisted generation.
- 28 MLLMs (general-purpose, science-oriented, chemistry-specialized, and embodied) were evaluated in an offline single-turn setting.
- Current models can follow coarse-grained experimental workflows in videos.
- However, they struggle with:
- grounding actions to actionable regions and feasible trajectories;
- recognizing fine-grained lab equipment, states, and safety norms;
- ordering medium-length procedural steps.
- BEAKER reveals a clear gap between coarse-grained process understanding and the precise, action-anchored perception required for laboratory execution, providing a diagnostic benchmark for future MLLMs in SDCLs.
Summary
Self-Driving Chemical Laboratories (SDCLs) are moving chemical experimentation toward embodied automation, where machines must perceive laboratory scenes, reason about experimental states, and act under physical, procedural, and safety constraints. Although Multimodal Large Language Models (MLLMs) are increasingly considered as embodied brains for such systems, existing laboratory benchmarks mainly focus on safety, anomaly detection, or general scientific understanding, leaving their execution-oriented embodied capabilities underexplored.
To fill this gap, the authors introduce BEAKER, a Benchmark for evaluating Embodied Actionability, Knowledge, Experimentation, and Reasoning in SDCLs.
Key Points
Findings
*Auto-collected on 2026-11-06.*