[论文] DeCAL: Towards Physically-Grounded Dexterous Vision-Language-Action Models via Contact-Aware Latent Co-Imagination

论文概要

研究领域: cs.RO, cs.AI
作者: Yankai Fu, Ning Chen, Junkai Zhao, Heng Zhang, Guocai Yao, Pengwei Wang, Zhongyuan Wang, Shanghang Zhang
发布时间: 2026-09-08
arXiv: 2609.09119

中文摘要

灵巧操作涉及与物理世界接触丰富且细粒度的交互，由于严重的视觉遮挡和复杂的接触动力学，对现有视觉-语言-动作（VLA）模型构成重大挑战。虽然近期工作已将触觉感知纳入机器人操作，但大多数方法仍依赖同质多模态融合，缺乏自适应触觉整合和对物理动力学的显式建模。在本工作中，我们提出DeCAL，一个物理基础的灵巧视觉-语言-动作模型，统一了接触丰富灵巧操作的理解、想象和动作生成。基于混合Transformer（MoT）架构构建，DeCAL为每种能力利用专门的专家，同时实现它们之间的高效信息流。为有效利用触觉信息，我们引入自适应视触融合，通过接触感知门控策略动态调节触觉交互。此外，我们提出视触潜在协同想象，联合建模视觉和触觉动力学，使策略具备隐式物理世界知识。实验结果表明，DeCAL在所有任务中始终达到最先进的性能，实现71%的平均成功率和83.4%的进度成功率，同时在未见场景中展现出强大的泛化能力。

原文摘要

Dexterous manipulation involves contact-rich and fine-grained interactions with the physical world, posing significant challenges for existing vision-language-action (VLA) models due to severe visual occlusions and complex contact dynamics. While recent works have incorporated tactile sensing into robotic manipulation, most approaches still rely on homogeneous multimodal fusion, lacking adaptive tactile integration and explicit modeling of physical dynamics. In this work, we present DeCAL, a physically-grounded dexterous vision-language-action model that unifies understanding, imagination and action generation for contact-rich dexterous manipulation. Built upon a Mixture-of-Transformers (MoT) architecture, DeCAL leverages specialized experts for each capability while enabling efficient information flow among them. To effectively leverage tactile information, we introduce Adaptive Visuo-Tactile Fusion that dynamically regulates tactile interactions via a contact-aware gating strategy. Furthermore, we propose Visuo-Tactile Latent Co-Imagination to jointly model visual and tactile dynamics, equipping the policy with implicit physical world knowledge. Experimental results show that DeCAL consistently achieves state-of-the-art performance across all tasks, attaining a 71% average success rate and an 83.4% progress success rate, while also demonstrating strong generalization to unseen scenarios.

自动采集于 2026-09-10

#论文 #arXiv #AI #小凯