NVIDIA SpatialClaw: Training-Free Agent with Code-as-Action Hits 59.9% on Spatial Reasoning

Forum topic · QianXun · 2026-06-22

Summary

NVIDIA Research has released SpatialClaw, a training-free agent framework for 3D spatial reasoning built around a novel 'code as action' interface. Instead of invoking perception tools through predefined JSON schemas, the agent writes Python code executed in a stateful Jupyter-style kernel, where tools like Depth Anything 3 and SAM 3 return ordinary variables (masks, depth maps, camera geometry, trajectories) that can be freely composed as NumPy arrays. Using the same Gemma4-31B backbone, prompts, and tools, SpatialClaw reaches 59.9% average accuracy across 20 benchmarks, versus 56.7% for structured tool-calling, 55.2% for single-pass code, and 53.4% for a no-tool baseline. The largest gains appear on dynamic tasks requiring chained geometric reasoning across frames and viewpoints (DSI-Bench +17.6, MindCube +15.3). An LLM-as-judge analysis attributes 52.2% of wins to code composition and 19.5% to control flow. The framework runs unchanged across six backbones (26B-397B) and is evaluated on robotics, multi-view detection, video/4D analysis, and indoor scene QA use cases.

NVIDIA Research introduced SpatialClaw (June 19, 2026), a training-free agent framework for spatial reasoning. Its central claim: the weakness of VLMs in 3D spatial judgment lies not in the model itself, but in the action interface the agent uses to call tools.

Paper: https://spatialclaw.github.io/static/pdfs/spatialclaw.pdf
Repository: https://github.com/NVlabs/SpatialClaw

Framework

SpatialClaw centers on a Stateful Python Kernel preloaded with input frames and a set of primitives:

Perception tools (Depth Anything 3, SAM 3) are ordinary Python callables
Outputs (masks, depth maps, camera geometry, trajectories) are ordinary Python variables
Six public entry points: InputImages, Metadata, tools, show(), vlm, ReturnAnswer()

Rather than calling tools via a predefined JSON schema, the agent writes a Python cell into the kernel, freely composes tool results as NumPy arrays, and decides the next step based on intermediate results.

Benchmark Results

Same Gemma4-31B backbone, same prompts, same perception tools — only the action interface changes:

| Action interface | Avg accuracy (20 benchmarks) | Δ vs no tools | |---|---|---| | No-tool baseline | 53.4% | — | | Single-pass code | 55.2% | +1.8 | | Structured tool-call (JSON schema) | 56.7% | +3.3 | | SpatialClaw (code as action) | 59.9% | +6.5 |

Compared to concurrent spatial agents:

| Method | Interface | Avg | Δ vs SpatialClaw | |---|---|---|---| | VADAR | Single-pass | 40.5 | −19.4 | | pySpatial | Single-pass | 47.8 | −12.1 | | SpaceTools-Toolshed | Structured tool-call | 48.7 | −11.2 | | SpatialClaw | Code as action | 59.9 | best |

An LLM-as-judge attribution analysis finds 52.2% of wins come from code composition, 19.5% from control flow, and 28.3% are interface-neutral.

Case study

Question: *What is the shortest distance between the radiator and the door?*

Single-pass code: writes a full program computing centroid distance using a median — wrong.
Structured tool-call: no "nearest point" operation in the tool schema — cannot answer.
SpatialClaw: first tries centroid distance, notices the issue, then switches to scipy.spatial.KDTree for nearest-point distance. Result: 0.9439 m (ground truth 0.9 m).

The key capability: deciding which tool to call and what code to write based on intermediate observations.

The largest gains occur on dynamic tasks requiring chained geometric reasoning across frames and viewpoints:

DSI-Bench: +17.6 points
MindCube: +15.3 points

Generalization and Implementation

The framework was validated on 6 backbones (Qwen3.5/3.6 and Gemma4 series, 26B–397B) with the same prompts and tool set, with no benchmark-specific tuning.

The five-stage loop: planning → code generation → code execution → feedback assembly → answer submission. AST static checks reject unsafe code; a 30-step cap applies.

Implementation stack: LangGraph workflow + persistent Jupyter kernel + vLLM-served backbones + FastAPI GPU services for perception.

Potential Applications

Robotics and embodied AI: measuring metric distances between objects before manipulation
Multi-view detection: recovering object orientation from multiple camera angles
Video and 4D analysis: tracking object or camera motion
Indoor scene QA: "Where is the door relative to the sink?"

Why It Matters

1. Paradigm shift: code-as-action vs JSON tool-calling marks a major turn in agent design — Anthropic's Computer Use, Claude Code, Manus, and Devin all share the "code is the action" core. 2. Training-free engineering value: one prompt/tool set works across 6 backbones and 20 benchmarks without retraining per VLM. 3. Upstream of VLA robots: SpatialClaw offers a training-free path for "seeing geometry first, then acting." 4. Perception is the ceiling reminder: with the action-interface lever nearly maxed (+6.5), better perception models are the next frontier.

Risks and Open Questions

Non-commercial license limits commercial deployment.
Perception quality remains the ceiling: accuracy may drop sharply with weaker backbones.
30-step cap may be hit on complex multi-step embodied tasks.
Code composition 52% vs JSON 28% implies JSON tool-calling still wins ~48% of scenarios.
Jupyter kernel + five-stage loop raises deployment cost compared to pure JSON tool-call pipelines.