GesVLA: A Gesture-Aware Vision-Language-Action Model for Robotic Manipulation

Forum topic · 小凯 · 2026-05-23

Summary

GesVLA is a gesture-aware vision-language-action (VLA) model proposed to address spatial ambiguity in robotic manipulation when text instructions alone cannot distinguish among multiple similar objects. The method encodes gesture features directly into the latent space, allowing them to participate in both high-level reasoning and low-level action generation, with a dual-VLM architecture tightly coupling gesture representation to the action policy. A scalable data pipeline renders hand models onto real-world scene images to reduce the sim-to-real visual gap and produce diverse pointing annotations. Training uses a two-stage strategy covering gesture perception and action prediction. Experiments on real-world tasks, including controlled block manipulation, product selection, and agricultural produce picking, show that incorporating gestures consistently improves target grounding accuracy and human-robot interaction efficiency, especially in complex, cluttered environments. Paper: arXiv 2505.17381.

GesVLA: Gesture-Aware Vision-Language-Action Model

Research area: Computer Vision Authors: Wenxuan Guo, Ziyuan Li, Meng Zhang Published: 2025-05-23 arXiv: 2505.17381

Overview

Vision-language-action (VLA) models show strong potential for general-purpose robotic manipulation by unifying perception and action. However, existing VLA systems rely mainly on text instructions and struggle to resolve spatial ambiguity in complex scenes where multiple similar objects are present.

To address this limitation, the authors introduce gestures as a parallel instruction modality and propose GesVLA, a gesture-aware vision-language-action model. Key contributions:

Latent gesture encoding: Gesture features are encoded directly into the latent space, enabling participation in both high-level reasoning and low-level action generation.
Dual-VLM architecture: A dual vision-language-model design tightly couples gesture representations with the action policy.
Scalable data pipeline: Hand models are rendered onto real-world scene images, reducing the sim-to-real visual gap while producing rich data with diverse motion patterns and corresponding pointing annotations.
Two-stage training: The model gains both gesture perception and action prediction capabilities through a two-stage training strategy.

Evaluation

The method was evaluated on multiple real-world robotic tasks, including a controlled block-manipulation task for validation, as well as more practical scenarios such as product selection and agricultural produce picking.

Results: Incorporating gestures consistently improves target grounding accuracy and human-robot interaction efficiency, particularly in complex and cluttered environments.