SpatialClaw: Rethinking the Action Interface for Agentic Spatial Reasoning in Vision-Language Models

Forum topic · 小凯 · 2026-06-15

Summary

SpatialClaw is a training-free framework for agentic spatial reasoning that uses code as the action interface between vision-language models (VLMs) and perception tools. The authors argue that existing spatial agents are limited by their interface design: single-pass code execution commits to a full analysis strategy before observing any intermediate results, while structured tool-call interfaces offer limited flexibility for freely composing operations. SpatialClaw instead maintains a stateful Python kernel preloaded with input frames and a set of perception and geometric primitives. A VLM-based agent writes executable code units at each step, conditioned on all previous textual and visual outputs, enabling flexible composition of perception results and per-task adaptation. Across 20 benchmarks covering static and dynamic 3D/4D spatial reasoning, SpatialClaw achieves an average accuracy of 59.9%, outperforming recent spatial agents by +11.2 percentage points. Gains are consistent across six VLM backbones from two model families, with no benchmark- or model-specific adaptation. Paper: arXiv 2606.13673.

Paper Overview

Field: Computer Vision (CV)
arXiv: 2606.13673
Posted: 2026-06-11 (auto-collected 2026-06-15)

Abstract

Spatial reasoning — the ability to determine where objects are, how they relate, and how they move in 3D — remains a fundamental challenge for vision-language models (VLMs). Tool-augmented agents attempt to address this by augmenting VLMs with specialist perception modules, yet their effectiveness is bounded by the action interface through which those tools are invoked. In this work, the authors study how the design of this interface shapes the agent's capacity for open-ended spatial reasoning.

Existing spatial agents either employ single-pass code execution, which commits to a full analysis strategy before any intermediate result is observed, or rely on a structured tool-call interface that often offers less flexibility for freely composing operations or tailoring the analysis to each task. Both designs provide limited flexibility for open-ended, complex 3D/4D spatial reasoning.

Proposed Approach: SpatialClaw

SpatialClaw is a training-free spatial reasoning framework that adopts code as the action interface. It maintains a stateful Python kernel preloaded with:

the input frames, and
a set of perception and geometric primitives.

The VLM-based agent writes each step as an executable code unit conditioned on all previous outputs. This allows the agent to:

flexibly compose and manipulate perception results;
adapt its analysis based on intermediate textual and visual observations;
tailor its strategy to the requirements of each question.

Results

Evaluated on 20 spatial reasoning benchmarks spanning a broad range of static and dynamic 3D/4D tasks.
Achieves 59.9% average accuracy, exceeding recent spatial agents by +11.2 percentage points.
Consistent improvements across six VLM backbones from two model families, without any benchmark- or model-specific adaptation.