[论文] Dreaming the Sound of Contact: Leveraging Video and Audio Generation f...

论文概要

研究领域: ML 作者: Guanhua Ji, Tianyu Li, Dayoon Suh, Yuqian Zhang, Boyan Zhang, Nadia Figueroa 发布时间: 2026-09-16 arXiv: 2609.19137

中文摘要

视频生成的最新进展使机器人能从生成的视频中学习操作轨迹。然而这些方法产生的是纯运动学轨迹，缺乏力信息，导致在接触密集型任务中失败——这类任务中适当的接触力是成功的关键。本工作探索用音频增强生成的视频：利用生成接触声音的响度来塑造有界的、时变的期望力曲线。我们提出一条流水线，联合利用生成的视频与音频，从结构化的自然语言任务提示中推导出运动轨迹和相应的期望力曲线。我们在 Franka Panda 机器人上用闭环力调节器执行这些力感知轨迹，在接触期间跟踪由音频塑造的力曲线。我们在多个需要接触的操纵任务上评估该流水线，证明了在纯运动学基线失败的情况下仍能成功操纵。我们还将该流水线用作数据生成引擎，训练出能以闭环方式完成任务的策略。项目网站、视频与数据集见项目页面。

原文摘要

Recent advances in video generation allow robots to learn manipulation trajectories from generated videos. However, these approaches produce purely kinematic trajectories that lack force information, causing failures in contact-rich tasks where appropriate contact forces are essential for success. In this work, we explore augmenting generated video with audio to shape a bounded, time-varying desired-force profile using the loudness of generated contact sounds. We present a pipeline that jointly leverages generated video and audio to derive motion trajectories and corresponding desired-force profiles from a structured natural-language task prompt. We execute these force-aware trajectories on a Franka Panda robot using a closed-loop force regulator that tracks the audio-shaped force profile ...

#论文 #arXiv #ML #小凯