Dreaming the Sound of Contact: Using Video and Audio Generation for Force-Aware Robot Manipulation

Forum topic · 小凯 · 2026-09-18

Summary

Recent advances in video generation allow robots to learn manipulation trajectories from generated videos, but these approaches produce purely kinematic trajectories lacking force information, leading to failures in contact-rich tasks where appropriate contact forces are essential. This work (arXiv:2609.19137) explores augmenting generated video with audio: the loudness of generated contact sounds is used to shape a bounded, time-varying desired-force profile. The authors present a pipeline that jointly leverages generated video and audio to derive both motion trajectories and corresponding desired-force profiles from a structured natural-language task prompt. These force-aware trajectories are executed on a Franka Panda robot using a closed-loop force regulator that tracks the audio-shaped force profile during contact. Evaluations across multiple contact-rich manipulation tasks show successful manipulation in cases where purely kinematic baselines fail. The pipeline is also used as a data-generation engine to train policies that complete tasks in a closed-loop manner. Project website, videos, and dataset are available on the project page.

Paper Overview

Field: Machine Learning (ML) Authors: Guanhua Ji, Tianyu Li, Dayoon Suh, Yuqian Zhang, Boyan Zhang, Nadia Figueroa Published: 2026-09-16 arXiv: 2609.19137

Abstract

Recent advances in video generation allow robots to learn manipulation trajectories from generated videos. However, these approaches produce purely kinematic trajectories that lack force information, causing failures in contact-rich tasks where appropriate contact forces are essential for success.

In this work, the authors explore augmenting generated video with audio to shape a bounded, time-varying desired-force profile using the loudness of generated contact sounds.

Key Contributions

A pipeline that jointly leverages generated video and audio to derive motion trajectories and corresponding desired-force profiles from a structured natural-language task prompt.
Execution of these force-aware trajectories on a Franka Panda robot using a closed-loop force regulator that tracks the audio-shaped force profile during contact.
Evaluation across multiple contact-rich manipulation tasks, demonstrating successful manipulation in cases where purely kinematic baselines fail.
Use of the pipeline as a data-generation engine to train policies capable of completing tasks in a closed-loop manner.

Project website, videos, and dataset are available on the project page.

--- *Auto-collected on 2026-09-18*