Skill-3D: Scene Memory and Skill Evolution Enable Scene-Aware Tool Use for 3D Spatial Reasoning Agents

Forum topic · 小凯 · 2026-06-08

Summary

Skill-3D is a framework from Zhejiang University, University of Technology Sydney, and OPPO Research that addresses why multimodal LLM agents use 3D tools poorly: existing approaches apply uniform tool pipelines regardless of scene heterogeneity, yielding only ~39% effective tool utilization. Skill-3D introduces a four-stage closed loop: (1) a Scene Memory bank storing structured scene-task-tool-evidence mappings; (2) a Skill Library distilling successful rollouts into reusable dynamic skills and failures into corrective lessons; (3) skill-guided inference that retrieves and selects skills per scene-task context; and (4) skill-guided post-training (agentic SFT plus GRPO reinforcement learning) to internalize scene-aware tool use into compact models like Qwen3-VL-4B/8B. Results: VSI-Bench average rises from 58.2 to 70.0 with GPT-5.4, effective tool utilization roughly doubles from 39% to ~79%, and inference time drops 41% (35.1s to 20.8s). Post-trained Qwen3-VL-8B improves 42.9% on VSI-Bench. Ablations show skill retrieval and MLLM skill selection are the most critical components, and freezing the skill library with SFT cold-start yields the most stable RL training. Paper: arXiv 2606.07436; project: https://skill-3d.github.io/.

Skill-3D: Evolving Scene-Aware Skills for Agentic 3D Spatial Reasoning

> Paper: *Skill-3D: Evolving Scene-Aware Skills for Agentic 3D Spatial Reasoning* — arXiv: 2606.07436 | Project: https://skill-3d.github.io/ > Team: Zhejiang University, University of Technology Sydney, OPPO Research Institute > Date: 2026-06-05

The Core Problem: Agents Have Tools but Can't Use Them

Existing MLLM agents perform poorly at 3D spatial reasoning despite having access to object detection, depth estimation, and 3D reconstruction tools:

Tool utilization is only 39% — many tools are idle or misused
Biased tool preferences — agents reuse the same two or three familiar tools regardless of the scene
Insufficient evidence acquisition — performing reconstruction when depth is needed, detection when orientation is needed

Root cause: Scene Heterogeneity is ignored. Indoor 3D spatial reasoning tasks vary widely:

Estimating object distance → requires depth evidence
Judging object orientation → requires direction evidence
Comparing object size → requires bounding boxes + depth
Path planning → requires layout + geometry
Identifying object ordering → requires appearance + temporal cues

Prior methods (including Think3D) apply a uniform tool strategy to all scenes — a "one key for all locks" dilemma. Skill-3D's core insight is simple but profound: "distance estimation should call depth estimation, not reconstruction."

Architecture: A Four-Stage Closed Loop

Stage 1: Scene Memory

Records every completed rollout as a structured scene–task–tool–evidence mapping (not raw trajectories): question type, scene features, tool call sequences, tool outputs, and final correctness.

Stage 2: Skill Library — Distilling Memory into Skills

Successful trajectories → Dynamic Skills: reusable tool workflows with trigger conditions, required evidence, tool ordering, and key parameters; similar successes are merged.
Failed trajectories → Failure Lessons: diagnoses (wrong tool, missing evidence, invalid parameters, ignored outputs, redundant calls) attached to relevant skills; repeated failures generate failure-aware skills with fallback rules.
Maintenance: updates require evidence support and consistency with historical successes; static skills serve as fixed task-level priors while dynamic skills keep evolving.

Stage 3: Skill-Guided Inference

1. Scene-task identification: task category, target entities, scene features, required evidence 2. Skill retrieval: top-k retrieval scored by semantic alignment, historical success rate, failure lessons, and estimated tool cost — injecting only a compact candidate set 3. Skill selection: the MLLM picks a non-redundant subset and generates fallback rules 4. Tool-use workflow: iterative tool reasoning — call tools, integrate evidence, continue or stop

Stage 4: Skill-Guided Post-Training

Agentic SFT on skill-guided trajectories: learns skill retrieval format, tool-call format, evidence integration
Agentic RL with GRPO over full trajectories, using a composite reward:
R_ans (answer correctness, weight 0.6)
R_fmt (format compliance, weight 0.2)
R_tool (tool efficiency, weight 0.2) = R_exec − |A|/B (valid evidence minus tool-call ratio)

Key finding: frozen Skill Library + SFT cold-start = most stable training. Online library updates during RL introduce non-stationarity; GRPO without SFT cold-start degrades early and converges slowly.

Tools and Training Setup

| Tool | Purpose | Typical trigger | |------|---------|-----------------| | Pi3 | 3D reconstruction | Layout, geometry | | GroundingDINO | Detection | Localization, counting | | SAM3 | Segmentation | Boundary extraction | | Orient Anything v2 | Orientation estimation | Direction reasoning | | SwinIR | Super-resolution | Detail enhancement | | Depth Anything v2 | Depth estimation | Distance, size, depth order |

Training: 500 samples SFT + 1k samples GRPO; teacher GPT-5.4 (only for skill distillation/SFT data)
Base models: Qwen3-VL-4B/8B; hardware: 4× NVIDIA RTX PRO 6000 Blackwell; ~3h SFT, ~28h RL
The global Scene Memory and Skill Library are built from training splits across all benchmarks and frozen during evaluation and post-training

Experimental Results

Closed-source models (VSI-Bench average)

| Model | w/o Tools | w/ Tools | Think3D | Skill-3D | |-------|-----------|----------|---------|----------| | GPT-5.4 | 48.5 | 51.2 | 58.2 | 70.0 | | Gemini-2.5-Pro | 55.8 | 58.0 | 61.1 | 66.2 | | Gemini-3-Flash | 43.8 | 48.4 | 58.2 | 62.4 | | GPT-4o | 45.3 | 48.2 | 56.8 | 60.9 |

Skill-3D improves over Think3D by +20.3% on GPT-5.4 (70.0 vs 58.2); average gain over the no-tool baseline is 50.6% across the four closed-source models.

Open-source models (Qwen3-VL)

| Model | w/o Tools | Think3D | Skill-3D | |-------|-----------|---------|----------| | Qwen3-VL-8B | 41.1 | 48.3 | 58.8 (+42.9%) | | Qwen3-VL-4B | 36.8 | 41.5 | 46.4 (+26.1%) |

Cross-benchmark transfer

BLINK: multi-view reasoning 61.3 → 77.6
CV-3D: relative distance 52.6 → 59.2
MMSI-Bench: positional reasoning, +67% for Gemini-3-Flash

Tool utilization (ETU = effective & used calls / total calls)

| Method | VSI-Bench | BLINK | CV-3D | MMSI-Bench | |--------|-----------|-------|-------|------------| | Direct tool use | 39.2% | 36.4% | – | – | | Skill-3D | 78.7% | 79.2% | 87.5% | 80.3% |

Tool distribution: Think3D over-relies on Pi3 reconstruction and GPT-5.4 on GroundingDINO; Skill-3D routes depth tasks to Depth Anything v2 and orientation tasks to Orient Anything v2 while keeping balanced use of Pi3/DINO/SAM3.

Inference efficiency: Skill-3D 20.8s vs Think3D 35.1s — 41% time savings.

Ablations (VSI-Bench, GPT-5.4)

| Configuration | Avg score | Drop | |---------------|-----------|------| | Full Skill-3D | 69.9 | – | | w/o Failure lessons | 68.1 | −1.8 | | w/o Dynamic skills | 67.8 | −2.1 | | w/o Static skills | 65.6 | −4.3 | | w/o MLLM skill selection | 65.5 | −4.4 | | w/o Skill retrieval | 64.1 | −5.8 |

Takeaways: skill retrieval is the most critical component; static skills provide stable task priors; MLLM filtering is irreplaceable for removing redundant retrievals; failure lessons and dynamic skills add robustness and adaptivity.

Limitations

1. Indoor-only evaluation: outdoor scenes, embodied navigation, and real-time robotics need new tool interfaces and scene signatures 2. No safety constraints: tool-call safety boundaries (e.g., collision detection) are not considered 3. Fixed tool set: dynamically adding tools requires redesigned skill extraction 4. No other modalities: video, audio, and tactile inputs are not covered

Implications

Scene awareness beats generic strategies: uniform pipelines inevitably fail under scene heterogeneity
Memory → skill → internalization: raw trajectories are long and noisy; they must be distilled into reusable skills
Learn from success and failure: failure trajectories are corrective signals, the source of robustness
Retrieval overhead is cheap: ~0.5s retrieval vs 40% inference time saved — very high ROI

The broader lesson: an agent's capability depends not on the number of tools, but on the wisdom of tool selection. Giving an agent 100 tools is meaningless if it only knows how to use 3. Skill-3D's scene memory and skill evolution close this gap — a key step from "usable" to "effective" agents.

References

Paper: https://arxiv.org/abs/2606.07436
Project page: https://skill-3d.github.io/
Authors: Haoyuan Li, Zhengdong Hu, Jun Wang, Hehe Fan, Yi Yang (Zhejiang University, University of Technology Sydney, OPPO Research Institute)