[论文] Pixel-UMI: Learning Environment-Aware Whole-Body Manipulation from Sen...

论文概要

研究领域: CV 作者: Jason Jingzhou Liu, Tony Tao, Deepak Pathak, Ruslan Salakhutdinov 发布时间: 2026-11-02 arXiv: 2611.03647

中文摘要

传感器化的人类示范为移动操作提供了可扩展的数据来源，但人与机器人之间的具身差异让策略学习变得困难。我们提出 Pixel-UMI——一套从人类数据中学习环境感知全身操作策略的管线。首先引入环境感知重定向（EAR）：结合重建的场景几何与示范者的头部和夹爪轨迹，推断机器人姿态并生成避碰的关节与基座目标；再通过延迟校准的动作分配将轨迹转为动作。然而运动学差异会残留姿态误差，导致人类示范与机器人观察之间存在视角错配。为此我们提出重定向误差感知掩蔽（REAM），在训练时利用这些误差，降低部署时对错位视角的依赖。最后，由于估计动作无法完全刻画富接触交互，我们通过 DAgger 收集在线人类纠正进一步提升性能。Pixel-UMI 在五个高难度任务上评估——涵盖铰接物体操作、可变形物体处理、受限空间伸展、全身避碰和富接触双手协调——平均任务进展达 87.0%。演示视频见 https://pixel-robot.com/pixel-umi

原文摘要

Sensorized human demonstrations offer scalable data for mobile manipulation, but embodiment differences complicate policy learning. We present Pixel-UMI, a pipeline for learning environment-aware whole-body policies from human data. We first introduce Environment-Aware Retargeting (EAR), which combines reconstructed scene geometry with demonstrated head and gripper trajectories to determine robot posture and generate collision-aware joint and base targets. Delay-calibrated action assignment converts these trajectories into actions. However, kinematic differences can leave residual pose errors, creating viewpoint mismatches between human demonstrations and robot observations. We therefore introduce Retargeting-Error-Aware Masking (REAM), which uses these errors during training to reduce rel...

#论文 #arXiv #CV #小凯