NVIDIA SpatialClaw: Code-as-Action Interface Achieves 59.9% Spatial Reasoning Accuracy Training-Free

Forum topic · QianXun · 2026-06-22

Summary

NVIDIA Research has released SpatialClaw, a training-free agent framework for 3D spatial reasoning built on a simple thesis: VLMs struggle with spatial judgment not because of the model, but because of the action interface agents use to call tools. SpatialClaw replaces rigid JSON-schema tool calls with a stateful Python kernel where perception tools (Depth Anything 3, SAM 3) are plain Python callables and outputs are ordinary variables that the model can freely compose. Using the same backbone, prompts, and tools, the code-as-action interface lifts average accuracy on 20 benchmarks from 53.4% (no tools) to 59.9%, beating single-pass code (55.2%), structured tool-calls (56.7%), and competing spatial agents like VADAR and pySpatial by 11-19 points. An LLM-as-judge analysis attributes 52.2% of wins to code composition and 19.5% to control flow. Gains are largest on dynamic multi-frame tasks (DSI-Bench +17.6, MindCube +15.3). The framework runs across six backbones (26B-397B) with no benchmark-specific tuning, uses a five-phase plan-code-execute-feedback-answer loop with AST safety checks and a 30-step limit, and points toward applications in robotics, multi-view detection, and video/4D analysis.

NVIDIA Research has released SpatialClaw, a training-free agent framework for spatial reasoning (June 19, 2026).

Core claim: The weakness of VLMs in 3D spatial judgment lies not in the model, but in the "action interface" through which the agent invokes tools.

Paper: https://spatialclaw.github.io/static/pdfs/spatialclaw.pdf Repository: https://github.com/NVlabs/SpatialClaw

How It Works

A Stateful Python Kernel comes pre-loaded with the input frames and a set of primitives.
Perception tools (Depth Anything 3, SAM 3) are ordinary Python callables.
Outputs (masks, depth maps, camera geometry, trajectories) are ordinary Python variables.
Six public entry points: InputImages, Metadata, tools, show(), vlm, ReturnAnswer().

Instead of calling tools via a predefined JSON schema, the agent writes a Python cell into the kernel, freely composes tool results as NumPy arrays, and decides the next step based on intermediate results.

Benchmark Results

Same Gemma4-31B backbone, same prompts, same perception tools — only the action interface changes:

| Action interface | Avg accuracy (20 benchmarks) | Δ vs no tools | |---|---|---| | No-tool baseline | 53.4% | — | | Single-pass code | 55.2% | +1.8 | | Structured tool-call (JSON schema) | 56.7% | +3.3 | | SpatialClaw (code as action) | 59.9% | +6.5 |

Compared with concurrent spatial agents:

| Method | Action interface | Avg | Δ vs SpatialClaw | |---|---|---|---| | VADAR | Single-pass | 40.5 | −19.4 | | pySpatial | Single-pass | 47.8 | −12.1 | | SpaceTools-Toolshed | Structured tool-call | 48.7 | −11.2 | | SpatialClaw | Code as action | 59.9 | Best |

An LLM-as-judge attribution analysis shows: 52.2% of wins come from code composition, 19.5% from control flow, and 28.3% are interface-neutral.

A concrete case from the paper:

> Q: What is the nearest distance between the radiator and the door? > > - Single-pass code: writes one program computing centroid distance (median as centroid). Wrong. > - Structured tool-call: the tool schema has no "nearest point" operation. Cannot answer. > - SpatialClaw: tries centroid distance first, notices it's wrong, then switches to scipy.spatial.KDTree for nearest points. 0.9439 m vs ground truth 0.9 m.

The key capability: deciding what tool to call and what code to write next, based on intermediate observations.

The largest gains occur on dynamic tasks:

DSI-Bench: +17.6 points
MindCube: +15.3 points

These require chained geometric computation across multiple frames and viewpoints — a loop that single-pass and JSON tool-call approaches cannot support.

The framework was validated on 6 backbones (Qwen3.5/3.6 and Gemma4 series, 26B to 397B) with the same prompts and tool set, no benchmark-specific tuning.

System Design

Five-phase loop: planning → code generation → code execution → feedback assembly → answer submission. AST static checks reject unsafe code. 30-step cap.

Stack: LangGraph workflow + persistent Jupyter kernel + vLLM-served backbones + FastAPI GPU services for perception.

Why It Matters

1. Paradigm shift: code-as-action vs JSON tool calls is a major turn in agent design — mirroring the philosophy behind Anthropic's Computer Use, Claude Code, Manus, and Devin. 2. Engineering value of training-free: one prompt/tool set works across 6 backbones and 20 benchmarks with no retraining per VLM. 3. Upstream for VLA robots: robots need to "see the geometry first" before physical manipulation — SpatialClaw offers a training-free path. 4. Perception is the ceiling reminder: the action-interface lever is nearly maxed (+6.5); the next gains require better perception models (Depth Anything 3, SAM 3).

Risks and Open Questions

Non-commercial license limits commercial deployment.
Perception is the ceiling: with weaker backbones, accuracy may collapse.
30-step cap may be hit on long-horizon embodied tasks (e.g., "open the drawer and take the cup").
52% code vs 28% neutral: JSON tool-calls still win in ~48% of scenarios — not fully replaced.
The five-phase loop + Jupyter kernel raises deployment cost compared to pure JSON tool-call pipelines.

Potential Applications

Robotics and embodied AI: measure metric distances between objects before manipulation.
Multi-view detection: recover object orientation from multiple camera angles.
Video and 4D analysis: track object or camera motion.
Indoor scene QA: "Where is the door relative to the sink?"