Pixel-UMI: Learning Environment-Aware Whole-Body Manipulation from Human Demonstrations

Forum topic · 小凯 · 2026-11-03

Summary

Pixel-UMI is a research pipeline (arXiv:2611.03647) by Jason Jingzhou Liu, Tony Tao, Deepak Pathak, and Ruslan Salakhutdinov that learns environment-aware whole-body robot manipulation policies from sensorized human demonstrations. The method addresses embodiment differences between humans and robots through three components: Environment-Aware Retargeting (EAR), which combines reconstructed scene geometry with demonstrated head and gripper trajectories to infer robot posture and generate collision-aware joint and base targets; Retargeting-Error-Aware Masking (REAM), which exploits residual retargeting pose errors during training to reduce reliance on mismatched viewpoints at deployment; and DAgger-based online human corrections to improve performance on rich-contact interactions where estimated actions are insufficient. Evaluated on five challenging tasks including articulated object manipulation, deformable object handling, reaching in confined spaces, whole-body collision avoidance, and rich-contact bimanual coordination, Pixel-UMI achieves 87.0% average task progress. Demo videos are available at https://pixel-robot.com/pixel-umi.

Paper Overview

Research Area: Computer Vision Authors: Jason Jingzhou Liu, Tony Tao, Deepak Pathak, Ruslan Salakhutdinov arXiv: 2611.03647

Abstract

Sensorized human demonstrations offer scalable data for mobile manipulation, but embodiment differences complicate policy learning. The authors present Pixel-UMI, a pipeline for learning environment-aware whole-body policies from human data.

Key Components

1. Environment-Aware Retargeting (EAR): Combines reconstructed scene geometry with demonstrated head and gripper trajectories to determine robot posture and generate collision-aware joint and base targets. Delay-calibrated action assignment then converts these trajectories into actions.

2. Retargeting-Error-Aware Masking (REAM): Kinematic differences leave residual pose errors, creating viewpoint mismatches between human demonstrations and robot observations. REAM uses these errors during training to reduce reliance on misaligned viewpoints at deployment.

3. DAgger Online Corrections: Since estimated actions cannot fully characterize rich-contact interactions, online human corrections are collected via DAgger to further improve performance.

Results

Pixel-UMI was evaluated on five challenging tasks:

Articulated object manipulation
Deformable object handling
Reaching in confined spaces
Whole-body collision avoidance
Rich-contact bimanual coordination

The system achieves 87.0% average task progress across these tasks.

Demonstration videos: https://pixel-robot.com/pixel-umi

--- *Auto-collected on 2026-11-04*