HumanCLAW: Evaluating Whether Vision-Language Models Can Act Through a Physical Body

Forum topic · 小凯 · 2026-07-31

Summary

HumanCLAW is an evaluation framework that decouples action decision-making from low-level motor execution when testing whether vision-language models (VLMs) can act through a physical body. At each step, a fixed, off-the-shelf VLM issues an atomic skill command that is translated into a sub-second continuous full-body motion chunk with real physical consequences including gravity and collisions. This isolates the model's action intelligence by removing execution-side disturbances, balance, and motor errors. The accompanying benchmark, HumanCLAW-Bench, contains 1,218 long-horizon, egocentric search-navigate-interact episodes across 41 indoor scenes. Tests on nine state-of-the-art VLMs showed none solved the benchmark, with the best achieving only 16.8% success rate. Target recognition is not the bottleneck; instead, VLMs lack embodied self-awareness, failing to track their own bodies, judge position, detect goal attainment, or notice collisions.

HumanCLAW: Can Vision-Language Models Act Through a Body?

Overview

Field: Computer Vision (CV)
Authors: Siyao Li, Jiawei Gu, Shuai Liu, Kairui Hu, Zekun Li, Linjie Li, Chengcheng Tang, Po-Chen Wu, Ivan Shugurov, Lingni Ma, Michael Zollhoefer, Sizhe An, Abhay Mittal, Amy Zhao, Ranjay Krishna, Manling Li, Ziwei Liu, Chuan Guo
Published: 2026-07-29
arXiv: 2607.27180

Summary

Evaluating whether a vision-language model (VLM) can act through a physical body is challenging. The outcome of an action couples the VLM's decision with motor control. When a task fails, it is hard to tell whether the VLM made a bad choice or the motor controller simply failed to execute it, for example by losing balance and falling.

In this work, the authors introduce HumanCLAW, an evaluation framework that decouples action decision-making from low-level execution. At every step, a harnessed, off-the-shelf VLM issues an atomic skill command, and the command is translated into a sub-second chunk of continuous full-body motion with real physical consequences, including gravity and collisions. The body can therefore act freely in the physical world, while execution-side disturbances, balance, and motor errors are excluded from the measurement. What remains measurable is the model's action intelligence: its immediate choice of what the body should do next.

Benchmark

Based on this framework, the authors construct HumanCLAW-Bench: 1,218 long-horizon, egocentric search-navigate-interact episodes across 41 indoor scenes.

Key Findings

Nine state-of-the-art VLMs were tested, and none solved the benchmark.
The best model reached only a 16.8% success rate.
Recognizing targets is not the bottleneck.
Current VLMs lack embodied self-awareness: they cannot track their own bodies, determine where they are, judge whether they have reached a target, or detect collisions with obstacles.

Implication

The work highlights that progress on embodied AI requires VLMs to develop explicit representations of their own physical state, rather than only improving object recognition or planning skills in the abstract.

--- *Auto-collected 2026-07-31*

#Paper #arXiv #CV

HumanCLAW: Evaluating Whether Vision-Language Models Can Act Through a Physical Body

Summary

HumanCLAW: Can Vision-Language Models Act Through a Body?

Overview

Summary

Benchmark

Key Findings

Implication

Tags