GROW²: Grounding Which and Where for Robot Tool Use

Forum topic · 小凯 · 2026-07-01

Summary

GROW² (GROunding Which and Where) is a robotics framework by Yuhong Deng, Yuyao Liu, and David Hsu that addresses open-world affordance grounding for creative robot tool use—selecting an open-category object as a tool (which) and localizing its action region (where). The method leverages object parts as a natural abstraction, splitting grounding hierarchically into semantic and geometric levels to avoid data-heavy end-to-end training. Semantically, a Vision-Language Model (VLM) parses natural-language task instructions, selects a suitable tool, and identifies task-relevant parts on both tool and target objects. Geometrically, visual foundation models ground the selected parts to precise 3D regions from a single RGB-D image. Experiments show GROW² outperforms state-of-the-art baselines on affordance prediction benchmarks, achieves zero-shot generalization to open-category objects, and surpasses baselines in both simulation and real-world robot tool-use tasks. Paper: arXiv 2507.00006.

Paper Overview

Field: Robotics Authors: Yuhong Deng, Yuyao Liu, David Hsu Published: 2026-07-01 arXiv: 2507.00006

Abstract

Can the robot use a plate to cut a cake if no knife is available? Tool use greatly expands robot capabilities, but to use tools creatively beyond their intended functions, the robot faces the challenge of open-world affordance grounding: select an open-category object to act as a tool and localize its specific region of action.

To this end, the authors introduce GROW² (GROunding Which and Where), which leverages object parts as a natural abstraction to split the grounding process hierarchically into semantic and geometric levels, thus bypassing the need for data-heavy, end-to-end training.

Semantic level: GROW² harnesses the commonsense reasoning of Vision-Language Models (VLMs) to parse a natural-language task instruction, select a suitable object as the tool, and identify task-relevant parts on the tool and the target object.
Geometric level: Visual foundation models then ground the selected parts from a single RGB-D image to precise 3D regions.

Results

Experiments on established benchmarks show that GROW² outperforms state-of-the-art baselines on affordance prediction. It also achieves zero-shot generalization on open-category objects and outperforms baselines in both simulation and real-world robot tool-use experiments.

---

*Auto-collected on 2026-07-01.*