Paper Overview
Field: Computer Vision / Robotics Authors: Haoran Yuan, Zekai Wang, Boning Shao, Haoran Lu, Trevor Darrell, Ismini Lourentzou, Wei Zhan arXiv: 2609.24976
Summary
Dexterous manipulation depends on contact dynamics that are often only partially observable from vision. Recent World-Action Models (WAMs) couple predictive video world modeling with action generation, but remain largely vision-centric and therefore cannot directly model these contact dynamics.
DexTacWAM addresses this by:
- Encoding tactile signals from each fingertip independently
- Aggregating features through a finger- and pose-aware tactile compressor
- Injecting tactile latents into a video diffusion world model for joint visuo-tactile world modeling
- Benchmark: Across six contact-rich dexterous manipulation tasks on a 22-DoF bimanual platform, DexTacWAM achieves the highest score on every task, averaging 70.6 versus 38.0 for the strongest baseline.
- Ablations: The gains come from modeling contact evolution as part of the predicted world state rather than using touch merely as a conditioning signal. Removing tactile world modeling drops a four-task mean from 74.7 to 26.6, even though the same tactile features and action expert are retained.
- Efficient adaptation: By fitting a tactile encoder for four hours on a frozen pretrained visual VAE, the approach extends a pretrained video model to touch with only ~100 demonstrations per task and no tactile pretraining, while visual prediction quality stays within 0.5 dB of the vision-only version.
- Compressor efficiency: The tactile compressor retains 89.4% of pre-fusion contact recall while achieving 2.26x faster training and 1.29x faster inference.
Key Results
Conclusion
These results indicate that pretrained video priors can be extended to distributed multi-fingertip contact dynamics in a data- and compute-efficient manner.
---
*Automatically collected on 2026-09-23 from zhichai.net.*