SpatialClaw: Rethinking Action Interface for Agentic Spatial Reasoning
- Field: Computer Vision
- arXiv: 2606.13673
- Authors: Seokju Cho, Ryo Hachiuma, Abhishek Badki, Hang Su, Byung-Kwan Lee, Chan Hee Song, Sifei Liu, Subhashree Radhakrishnan, Seungryong Kim, Yu-Chiang Frank Wang, Min-Hung Chen
- Flexibly compose and manipulate perception results
- Adapt its analysis based on intermediate textual and visual observations
- Tailor the analysis to the needs of each question
- Evaluated on 20 spatial reasoning benchmarks spanning a broad range of static and dynamic 3D/4D tasks
- Achieves 59.9% average accuracy, surpassing recent spatial agents by +11.2 percentage points
- Consistent improvements across six VLM backbones from two model families
- Requires no benchmark-specific or model-specific adaptation
Summary
Spatial reasoning — the ability to determine where objects are, how they relate, and how they move in 3D — remains a fundamental challenge for vision-language models (VLMs). Tool-augmented agents attempt to address this by augmenting VLMs with specialist perception modules, yet their effectiveness is bounded by the action interface through which those tools are invoked.
This work studies how the design of that interface shapes an agent's capacity for open-ended spatial reasoning. Existing spatial agents either employ single-pass code execution, which commits to a full analysis strategy before any intermediate result is observed, or rely on a structured tool-call interface, which offers less flexibility for freely composing operations or tailoring the analysis to each task. Both designs provide limited flexibility for open-ended, complex 3D/4D spatial reasoning.
Approach
The authors propose SpatialClaw, a training-free spatial reasoning framework that uses code as the action interface. SpatialClaw maintains a stateful Python kernel preloaded with the input frames and a set of perception and geometric primitives. A VLM-powered agent writes an executable code unit for each step, conditioned on all previously produced outputs. This enables the agent to:
Results
Original Abstract (excerpt)
> Spatial reasoning, the ability to determine where objects are, how they relate, and how they move in 3D, remains a fundamental challenge for vision-language models (VLMs). Tool-augmented agents attempt to address this by augmenting VLMs with specialist perception modules, yet their effectiveness is bounded by the action interface through which those tools are invoked...
Full abstract: arXiv:2606.13673