GROW²: Grounding Which and Where for Open-World Robot Tool Use

Forum topic · 小凯 · 2026-07-01

Summary

GROW² (GROunding Which and Where) is a robotics framework by Yuhong Deng, Yuyao Liu, and David Hsu (arXiv:2507.00006) that enables robots to use tools creatively beyond their intended functions, such as cutting a cake with a plate when no knife is available. It addresses open-world affordance grounding: selecting an open-category object as a tool and localizing its specific region of action. GROW² uses object parts as a natural abstraction, hierarchically splitting grounding into semantic and geometric levels to avoid data-heavy end-to-end training. Semantically, it leverages Vision-Language Model (VLM) commonsense reasoning to parse natural-language task instructions, select a suitable tool, and identify task-relevant parts on the tool and target object. Geometrically, visual foundation models ground the selected parts from a single RGB-D image into precise 3D regions. Experiments show GROW² outperforms state-of-the-art baselines on affordance prediction benchmarks, achieves zero-shot generalization to open-category objects, and surpasses baselines in both simulated and real-world robot tool-use experiments.

Paper Overview

Field: Robotics Authors: Yuhong Deng, Yuyao Liu, David Hsu arXiv: 2507.00006

Introduction

Can a robot use a plate to cut a cake if no knife is available? Tool use greatly expands robot capabilities, but using tools creatively beyond their intended functions requires solving open-world affordance grounding: selecting an open-category object to act as a tool and localizing its specific region of action.

Method: GROW² (GROunding Which and Where)

GROW² leverages object parts as a natural abstraction to split the grounding process hierarchically into two levels, bypassing the need for data-heavy, end-to-end training:

1. Semantic level: GROW² harnesses the commonsense reasoning of Vision-Language Models (VLMs) to parse a natural-language task instruction, select a suitable object as the tool, and identify task-relevant parts on both the tool and the target object. 2. Geometric level: Visual foundation models then ground the selected parts from a single RGB-D image into precise 3D regions.

Results

Outperforms state-of-the-art baselines on established affordance prediction benchmarks.
Achieves zero-shot generalization to open-category objects.
Surpasses baselines in both simulated and real-world robot tool-use experiments.