English static mirror for SEO/GEO · AI-assisted translation · Read Chinese original

GesVLA: Gesture-Aware Vision-Language-Action Model for Robot Manipulation

Forum topic · 小凯 · 2026-05-25

Summary

GesVLA is a gesture-aware vision-language-action (VLA) model introduced to address spatial ambiguity in robot manipulation scenes containing multiple similar objects, where text-only instructions fall short. The approach encodes hand gesture features directly into the latent space, allowing them to participate in both high-level reasoning and low-level action generation, using a dual-VLM architecture that tightly couples gesture representations with the action policy. To overcome data scarcity, the authors build a scalable gesture data pipeline by rendering hand models onto real scene images, reducing the sim-to-real visual gap and producing diverse motion patterns with pointing annotations. A two-stage training strategy equips the model with gesture awareness and action prediction. Experiments on real-world robot tasks, including controlled block manipulation and practical product and produce selection scenarios, show that incorporating gestures consistently improves target grounding accuracy and human-robot interaction efficiency, especially in cluttered environments. Paper: arXiv 2505.14489.

Overview

  • Field: Computer Vision (CV)
  • Authors: Wenxuan Guo, Ziyuan Li, Meng Zhang
  • Published: 2026-05-25
  • arXiv: 2505.14489
  • Key Points

    Vision-language-action (VLA) models show strong potential for generalist robot manipulation by unifying perception and action. However, existing VLA systems rely mainly on text instructions and struggle with spatial ambiguity in complex scenes containing multiple similar objects.

    To address this limitation, the authors introduce gestures as a parallel instruction modality and propose GesVLA, a gesture-aware vision-language-action model:

  • Latent gesture encoding: Hand gesture features are encoded directly into the latent space, enabling them to participate in both high-level reasoning and low-level action generation.
  • Dual-VLM architecture: A dual-VLM design tightly couples gesture representations with the action policy.
  • Scalable data pipeline: Gesture data is generated by rendering hand models onto real scene images, reducing the sim-to-real visual gap while producing rich data with diverse motion patterns and corresponding pointing annotations.
  • Two-stage training: A two-stage training strategy endows the model with gesture awareness and action prediction capabilities.
  • Evaluation

    The method was evaluated on multiple real-world robot tasks, including:

  • A controlled block manipulation task for validation
  • More practical scenarios such as product and agricultural produce selection
Experimental results show that incorporating gestures consistently improves target grounding accuracy and human-robot interaction efficiency, particularly in complex and cluttered environments.

---

*Auto-collected on 2026-05-25. Source: arXiv:2505.14489*

Tags

#gesvla#vision-language-action#robot-manipulation#gesture-recognition#computer-vision#human-robot-interaction#sim-to-real#arxiv

This page is an English static mirror generated for search and AI citation. It may be a full translation or structured summary of the Chinese original. Canonical interactive discussion lives on the Chinese page: https://zhichai.net/topic/177620762