SpatialClaw: Rethinking Action Interface for Agentic Spatial Reasoning
A detailed walkthrough of the SpatialClaw paper (NVIDIA Research + KAIST, arXiv:2606.13673), explaining why state-of-the-art VLMs still stumble on questions humans find trivial — "Is that car driving toward the camera?" — and how an iterative code-interaction interface solves it.
The Problem
Ask a VLM: "How far is the white radiator from the door in this room?" Humans glance and estimate ~1.5 meters. VLMs answer absurdly (6.5m), or confidently wrong with false precision (1.638m). Spatial reasoning is not a native language-model capability: mapping pixels to 3D distances requires geometry, multi-view consistency, and temporal reasoning.
Two Dead Ends in Prior Work
- Single-shot code execution (pySpatial, VADAR): the VLM writes a full program up front. If segmentation fails mid-run, there is no chance to adjust — the program commits to a complete strategy before executing.
- Structured tool calling (SpaceTools): fixed JSON APIs (
get_depth(),segment()). If the model needs an operation the API designer never anticipated (e.g., nearest-neighbor queries via KDTree instead of centroids), it is stuck. - Stage I — Planning: a separate text-only LLM session (no image input) produces a high-level strategy and tool plan, appended to the agent's system prompt.
- Stage II — Code generation: the VLM must emit a structured markdown cell: Purpose / Reasoning / Next Goal / Code.
- Stage III — Execution: AST-based sandbox rejects
exec,eval, file I/O, and network access; regex checks catch.save(),.to_csv(); persistent kernel runs the code. Rejections return reasons, enabling same-step fixes. - Stage IV — Feedback assembly: stdout, variable summaries, visualizations, and tracebacks are appended to context.
- Stage V — Answer submission:
ReturnAnswer()with format validation; on failure, loop continues or a CoT fallback with regex extraction terminates. tools.Reconstruct— Depth Anything 3 (DA3Nested-Giant-Large): per-frame depth, camera intrinsics/extrinsics, dense point clouds with real-world metric scale.tools.SAM3— segmentation with text/point/box prompts.- Additional namespaces:
tools.Geometry,tools.Mask,tools.Time,tools.Graph,tools.Draw. - 59.9% average across 20 benchmarks with Gemma4-31B vs 53.4% without tools (+6.5pp), and +11.2pp over the best prior spatial agents. Largest gains in multi-view (+13.6) and video/4D (+10.4) reasoning.
- Head-to-head interfaces (20-benchmark average): no tools 53.4%, single-shot code 55.2%, structured tools 56.7%, SpatialClaw 59.9% (+4.7pp vs single-shot, +7.9pp vs structured tools).
- Most counterintuitive ablation: removing all predefined tool helper functions costs only 0.5pp (56.9% → 56.4%). The persistent kernel plus numpy/scipy alone compensates for nearly everything. Removing perception models costs 5.5pp.
- Attribution: 52.2% of gains come from code composition, 19.5% from control flow — over half the improvement comes from *combining* tools in code, not from any single tool.
- Dual-role serving: vLLM with OpenAI-compatible API and prefix caching for all LLM calls; lightweight HTTP services for DA3 and SAM3.
- Frame-type contracts: frame-index-related outputs are wrapped in typed containers validated at composition time, preventing cross-frame semantic errors.
- Tiered error handling: format errors → sandbox rejection (same-step retry) → trimmed traceback → cell timeout (namespace reset) → perception-tool retries with exponential backoff → forced termination via CoT fallback.
- Robust computation: median over mean, multi-frame verification, check units and orders of magnitude, print and verify intermediate values.
- Cross-validation: visual perception, geometry, visualization, and logical reasoning as four evidence sources.
- Five-step diagnosis: segmentation quality → depth plausibility → coordinate consistency → numeric ranges → multi-evidence alignment.
- Cho, S., et al. (2026). "SpatialClaw: Rethinking Action Interface for Agentic Spatial Reasoning." arXiv:2606.13673.
- Project page: https://nvlabs.github.io/SpatialClaw
- NVIDIA Research
Shared flaws: no intermediate verification, no on-the-fly correction, no free composition with scientific libraries.
Core Idea: Code as an "Orchestration Space"
SpatialClaw treats Python not as a one-shot program or an API dispatch layer, but as a persistent sandbox for composing, diagnosing, and revising perception tools — write one step, observe, fix, continue.
Key mechanisms:
1. State persistence: a persistent Python kernel keeps variables across steps (seg_heater from step 1 is still available in step 4).
2. Rich intermediate feedback: stdout, variable summaries (type/length/shape), registered visualizations via show() embedded into context, and trimmed tracebacks — the model can literally *see* its segmentation result and redo it.
3. Free composition: tool outputs are plain numpy arrays, usable with scipy (KDTree, ConvexHull), matplotlib, and normal control flow. No boundary between tool outputs and native Python objects.
4. Instant correction: mid-loop strategy switches, e.g., replacing centroid-based distance with KDTree nearest-neighbor queries.
Architecture: Five-Stage Loop
Entry Points and Tools
Six public entry points: InputImages (up to 32 frames, ≤768px long edge), Metadata (FPS, duration, frame indices), tools, show(...), vlm (a separate VLM session with vlm.locate() and vlm.ask_with_thinking() as an on-demand "advisor"), and ReturnAnswer(...).
Perception layer:
Key Results
Engineering Details
Design Philosophy
The system prompt encodes general meta-cognitive principles, not task-specific instructions:
The Bigger Picture
SpatialClaw argues for a paradigm shift: from "tools as APIs" (capability boundaries defined by the designer) to "code as the action interface" (boundaries defined by the language itself). Ablations confirm that what limits agents is not a lack of tools but a lack of freedom in interaction. The paradigm extends naturally to data science, robot control, and scientific experimentation — any domain requiring trial, verification, and correction.