Skill-3D: Scene-Aware Skill Evolution Closes the Loop for Agentic 3D Spatial Reasoning
> Paper: *Skill-3D: Evolving Scene-Aware Skills for Agentic 3D Spatial Reasoning* > arXiv: 2606.07436 | Project: https://skill-3d.github.io/ > Team: Zhejiang University, University of Technology Sydney, OPPO Research > Date: 2026-06-05
The Core Problem: Agents Have Tools but Don't Use Them Well
Existing MLLM agents perform poorly on 3D spatial reasoning despite having access to object detection, depth estimation, and 3D reconstruction tools:
- Effective tool utilization is only 39% — many tools sit idle or get misused
- Biased tool preference — agents stick to two or three familiar tools regardless of the scene
- Insufficient evidence acquisition — reconstructing when depth is needed, detecting when orientation is needed
- Estimating object distance → needs depth evidence
- Judging object orientation → needs direction evidence
- Comparing object size → needs boundaries + depth
- Path planning → needs layout + geometry
- Ordering objects → needs appearance + temporal cues
- "How far is the table from the stove?" → depth estimation only, no 3D reconstruction
- "Is the sofa bigger than the chair?" → detection + boundaries + depth, no orientation estimation
- "How do I get from the door to the kitchen?" → layout + geometry, no appearance analysis
- Successful trajectories → Dynamic Skills: reusable tool workflows with trigger conditions, required evidence, tool order, and parameters; similar successes are merged to broaden coverage.
- Failed trajectories → Failure Lessons: diagnoses (wrong tool, missing evidence, invalid parameters, ignored outputs, redundant calls) attached to related skills; repeated failures produce failure-aware skills with fallback rules.
- Maintenance: new updates require supporting evidence and consistency with past successes. Static skills are fixed task-level priors; dynamic skills keep evolving.
- Agentic SFT: training on skill-guided trajectories to learn skill retrieval, tool call formats, and evidence integration
- Agentic RL: GRPO with a composite reward — R_ans (answer correctness, 0.6), R_fmt (format, 0.2), R_tool (tool efficiency, 0.2), where R_tool = R_exec − |A|/B penalizes redundant calls
- Key finding: frozen Skill Library + SFT cold start is the most stable configuration; online skill-library updates introduce non-stationarity and degrade GRPO training
- BLINK: multi-view reasoning 61.3 → 77.6
- CV-3D: relative distance 52.6 → 59.2
- MMSI-Bench: Gemini-3-Flash improved 67% on positional reasoning
- Paper: https://arxiv.org/abs/2606.07436
- Project page: https://skill-3d.github.io/
- Authors: Haoyuan Li, Zhengdong Hu, Jun Wang, Hehe Fan, Yi Yang (Zhejiang University, UTS, OPPO Research)
Root cause: scene heterogeneity is ignored. Indoor 3D spatial reasoning tasks differ fundamentally:
Prior methods (including Think3D) apply a uniform tool policy across all scenes — a "one key opens every lock" problem.
Core Insight: Different Scenes Need Heterogeneous Tool Flows
> "Distance estimation should call the depth tool, not reconstruction."
Architecture: A Four-Stage Closed Loop
Stage 1: Scene Memory
Stores structured scene–task–tool–evidence mappings from completed rollouts (question type, scene features, tool call sequences, tool outputs, final answers) — not raw trajectories.
Stage 2: Skill Library — Distilling Memory into Skills
Stage 3: Skill-Guided Inference
1. Scene-task recognition: task category, target entities, scene features, required evidence 2. Skill retrieval: top-k retrieval scored by semantic alignment, historical success rate, failure lessons, and tool cost — returns a compact candidate set, not the whole library 3. Skill selection: the MLLM picks a non-redundant subset and generates fallback rules (e.g., detection → segmentation when boundaries are needed) 4. Tool-use workflow: iterative tool reasoning with evidence integration until answering
Stage 4: Skill-Guided Post-Training
Tools and Training Setup
| Tool | Purpose | Typical trigger | |------|---------|----------------| | Pi3 | 3D reconstruction | layout, geometry | | GroundingDINO | detection | localization, counting | | SAM3 | segmentation | boundaries | | Orient Anything v2 | orientation | direction reasoning | | SwinIR | super-resolution | small objects, detail | | Depth Anything v2 | depth | distance, size, order |
Training: 500 SFT samples + 1k GRPO samples; teacher GPT-5.4 (only for skill distillation/SFT data); base model Qwen3-VL-4B/8B; hardware 4× NVIDIA RTX PRO 6000 Blackwell; ~3h SFT, ~28h RL.
Experimental Results
Closed-source models (VSI-Bench average)
| Model | w/o Tools | w/ Tools | Think3D | Skill-3D | |-------|-----------|----------|---------|----------| | GPT-5.4 | 48.5 | 51.2 | 58.2 | 70.0 | | Gemini-2.5-Pro | 55.8 | 58.0 | 61.1 | 66.2 | | Gemini-3-Flash | 43.8 | 48.4 | 58.2 | 62.4 | | GPT-4o | 45.3 | 48.2 | 56.8 | 60.9 |
Open-source models (Qwen3-VL)
| Model | w/o Tools | w/ Tools | Think3D | Skill-3D | |-------|-----------|----------|---------|----------| | Qwen3-VL-8B | 41.1 | 44.7 | 48.3 | 58.8 | | Qwen3-VL-4B | 36.8 | 38.4 | 41.5 | 46.4 |
Cross-benchmark transfer
Effective Tool Utilization (ETU)
| Method | VSI-Bench | BLINK | CV-3D | MMSI-Bench | |--------|-----------|-------|-------|------------| | Direct tool use | 39.2% | 36.4% | – | – | | Skill-3D | 78.7% | 79.2% | 87.5% | 80.3% |
Tool distribution analysis: Think3D over-relies on Pi3 reconstruction and GPT-5.4 on GroundingDINO detection; Skill-3D appropriately increases Depth Anything v2 for depth tasks and Orient Anything v2 for orientation tasks.
Inference time: Skill-3D 20.8s vs Think3D 35.1s — 41% faster.
Ablations (VSI-Bench, GPT-5.4)
| Configuration | Score | Drop | |---------------|-------|------| | Full Skill-3D | 69.9 | – | | w/o failure lessons | 68.1 | −1.8 | | w/o dynamic skills | 67.8 | −2.1 | | w/o static skills | 65.6 | −4.3 | | w/o MLLM skill selection | 65.5 | −4.4 | | w/o skill retrieval | 64.1 | −5.8 |
Findings: skill retrieval is the most critical component; static skills provide stable priors; MLLM selection is irreplaceable for filtering redundant top-k candidates; failure lessons add robustness.
For GRPO training, frozen skill library + SFT cold start yields the most stable training and highest final reward; online library updates and no cold start both underperform.
Comparison with Existing Methods
| Dimension | No-agent baseline | Direct tool use | Think3D | Skill-3D | |-----------|------------------|-----------------|---------|----------| | Tool policy | none | uniform | uniform reconstruction | scene-aware skills | | Tool utilization | 0% | ~39% | ~39% | ~79% | | Inference time | shortest | medium | 35.1s | 20.8s | | Scene adaptation | no | no | no | yes (dynamic skills) | | Learning from failure | no | no | no | yes (failure lessons) |
Limitations
1. Indoor focus: outdoor scenes, embodied navigation, and real-time robot interaction need new tool interfaces and scene signatures 2. Safety constraints: tool-call safety boundaries (e.g., collision detection) are not considered 3. Fixed toolset: adding new tools requires redesigning skill extraction 4. Modalities: video, audio, and tactile inputs are not covered
Key Takeaways
1. Scene awareness beats generic policies — uniform workflows fail on heterogeneous scenes 2. Memory → skill → internalization — raw trajectories must be distilled into reusable skills 3. Learn from both success and failure — failed trajectories are corrective signals, not garbage 4. Retrieval overhead is acceptable — ~0.5s retrieval vs 40% inference-time savings
> An agent's capability is not determined by how many tools it has, but by the wisdom of tool selection.
Conclusion
Skill-3D achieves: tool utilization doubled (39% → 79%), VSI-Bench up from 58.2 to 70.0, 40% faster inference, effective cross-benchmark transfer with a single frozen skill library, and deployable compact models (Qwen3-VL-8B +43% after post-training). It marks a paradigm shift for agent design: from stacking tools to using them precisely, from uniform strategies to scene awareness, from raw trajectories to evolving skills.