Paper Overview
- Field: Machine Learning
- Authors: Guanhua Ji, Tianyu Li, Dayoon Suh, Yuqian Zhang, Boyan Zhang, Nadia Figueroa
- Published: 2026-09-16
- arXiv: 2609.19137
- Uses generated contact-sound loudness to define time-varying, bounded desired-force profiles
- Joint video+audio pipeline conditioned on structured natural-language task prompts
- Closed-loop force regulation on real hardware (Franka Panda)
- Demonstrates success on contact-rich tasks where kinematic-only baselines fail
- Doubles as a data-generation engine for training closed-loop manipulation policies
Summary
Recent advances in video generation allow robots to learn manipulation trajectories from generated videos. However, these approaches produce purely kinematic trajectories that lack force information, causing failures in contact-rich tasks where appropriate contact forces are essential for success.
This work explores augmenting generated video with audio to shape a bounded, time-varying desired-force profile using the loudness of generated contact sounds. The authors present a pipeline that jointly leverages generated video and audio to derive both motion trajectories and corresponding desired-force profiles from a structured natural-language task prompt.
These force-aware trajectories are executed on a Franka Panda robot using a closed-loop force regulator that tracks the audio-shaped force profile during contact. Evaluations across multiple contact-rich manipulation tasks demonstrate successful manipulation in cases where purely kinematic baselines fail. The pipeline is further used as a data-generation engine to train policies that complete tasks in a closed-loop manner.
Key Contributions
Original Abstract (excerpt)
> Recent advances in video generation allow robots to learn manipulation trajectories from generated videos. However, these approaches produce purely kinematic trajectories that lack force information, causing failures in contact-rich tasks where appropriate contact forces are essential for success. In this work, we explore augmenting generated video with audio to shape a bounded, time-varying desired-force profile using the loudness of generated contact sounds. We present a pipeline that jointly leverages generated video and audio to derive motion trajectories and corresponding desired-force profiles from a structured natural-language task prompt. We execute these force-aware trajectories on a Franka Panda robot using a closed-loop force regulator that tracks the audio-shaped force profile ...
*Auto-collected on 2026-09-18*