[论文] HumanCLAW: Can Vision-Language Models Act Through a Body?

论文概要

研究领域: CV
作者: Siyao Li, Jiawei Gu, Shuai Liu, Kairui Hu, Zekun Li, Linjie Li, Chengcheng Tang, Po-Chen Wu, Ivan Shugurov, Lingni Ma, Michael Zollhoefer, Sizhe An, Abhay Mittal, Amy Zhao, Ranjay Krishna, Manling Li, Ziwei Liu, Chuan Guo
发布时间: 2026-07-29
arXiv: 2607.27180

中文摘要

评估视觉-语言模型（VLM）能否通过物理身体行动是一个挑战。动作的结果将VLM的决策与运动控制耦合在一起。当任务失败时，很难判断是VLM做出了错误选择，还是运动控制器未能执行，例如失去平衡摔倒。本文提出HumanCLAW，一个将动作决策与低级执行解耦的评估框架。每一步，一个被固定住的开箱即用VLM发出原子技能指令，指令被转换为亚秒级的连续全身运动块，具有真实的物理后果，包括重力和碰撞。因此身体可以在物理世界中自由行动，而执行端的扰动、平衡和运动误差被排除在外。剩下可测量的是模型的动作智能：它对身体下一步应该执行什么的即时选择。基于该框架，我们构建了HumanCLAW-Bench：41个室内场景中的1,218个长程、以自我为中心的寻找-导航-交互片段。我们测试了九个最先进的VLM，发现没有一个能解决该基准；最佳模型仅达到16.8%的成功率。识别目标不是瓶颈。当前VLM缺乏的是具身自我意识：它们无法跟踪自己的身体，无法判断自己在哪、是否到达目标、是否撞到了障碍物。

原文摘要

Evaluating whether a vision-language model (VLM) can act through a physical body is challenging. The outcome of an action couples the VLM's decision with motor control. When a task失败, it is hard to tell whether the VLM made a bad choice or the motor controller simply failed to execute it, e.g., losing balance and falling. In this work, we introduce HumanCLAW, an evaluation framework that decouples action decision-making from low-level execution. At every step, a harnessed, off-the-shelf VLM issues an atomic skill command, and the command is translated into a sub-second chunk of continuous full-body motion with real physical consequences, including gravity and collisions. The body can therefore act freely in the physical world, while execution-side disturbances, balance and motor errors, ar...

自动采集于 2026-07-31

#论文 #arXiv #CV #小凯