Sensorized human demonstrations offer scalable data for mobile manipulation, but embodiment differences complicate policy learning. We present Pixel-UMI, a pipeline for learning environment-aware whole-body policies from human data. We first introduce Environment-Aware Retargeting (EAR), which combines reconstructed scene geometry with demonstrated head and gripper trajectories to determine robot posture and generate collision-aware joint and base targets. Delay-calibrated action assignment converts these trajectories into actions. However, kinematic differences can leave residual pose errors, creating viewpoint mismatches between human demonstrations and robot observations. We therefore introduce Retargeting-Error-Aware Masking (REAM), which uses these errors during training to reduce reliance on mismatched views at deployment. Finally, because estimated actions cannot fully capture contact-rich interactions, we collect on-policy human corrections through DAgger to improve performance. We evaluate Pixel-UMI on five challenging tasks spanning articulated-object manipulation, deformable-object handling, constrained-space reaching, whole-body collision avoidance, and contact-rich bimanual coordination, achieving 87.0% average task progress.
System Design
Pixel-UMI
We co-design Pixel-UMI with Pixel to align gripper geometry, camera viewpoints, and tracked motion. The wearable system captures human demonstrations together with scene depth, providing the motion and geometry needed for environment-aware retargeting.
Loading scene…
Reconstruction
We reconstruct the surrounding scene directly from depth captured during each demonstration, without a separate scene scan. Combining observations over time helps separate static obstacles from moving hands and manipulated objects.
Loading scene…
Scene 1 · Hang Jacket
Loading scene…
Scene 2 · Pillow
Loading scene…
Scene 3 · Pack Tennis Canister
Loading scene…
Scene 4 · Fridge
Loading scene…
Scene 5 · Mouse
Loading scene…
Scene 6 · Box
Retargeting
Environment-Aware Retargeting
We use reconstructed scene geometry to turn human head and gripper motion into coordinated whole-body robot trajectories. Our retargeting scheme teaches the policy how to position its arms, torso, and base to complete the task while avoiding nearby obstacles.
Pack Tennis Canister
Loading scene…
With Environment-Aware Retargeting
Loading scene…
Without Environment-Aware Retargeting
With Environment-Aware RetargetingWithout Environment-Aware Retargeting
With Environment-Aware RetargetingWithout Environment-Aware Retargeting
With Environment-Aware RetargetingWithout Environment-Aware Retargeting
With Environment-Aware RetargetingWithout Environment-Aware Retargeting
With Environment-Aware RetargetingWithout Environment-Aware Retargeting
Action Assignment
Human demonstrations record state, but reproducing that motion requires robot actions. We account for each joint’s controller characteristics when assigning action targets before training, helping the robot follow the demonstration more closely.
Observe
Find the delay
Apply it
Action assignment step: Observe
Policy Learning
Dynamic Camera Masking
The robot cannot always reproduce the human’s camera viewpoints due to kinematic mismatches. During training, we increase the probability of masking views with larger retargeting errors, reducing reliance on observations that may not transfer well to the robot.
Loading scene…
DAgger Intervention
For challenging tasks, human demonstrations provide an initial policy, while corrections on the robot help address gaps in the policy performance. An operator uses the VR interface to intervene during rollouts, producing robot observation–command pairs for further training.
Fold Box
Hang Jacket
Autonomous Tasks
Human Data Only
Policies can learn diverse and nuanced behaviors from human demonstrations without any robot demonstration data.
Autonomous 1x
Pack Tennis Canister
Pixel learns the arm posture needed to reach into confined spaces.
Autonomous 1x
Retrieve Drink
Pixel can operate articulated objects and use its full range of motion to reach into the fridge.
Autonomous 1x
Arrange Mouse
Pixel learns to adapt its behavior around obstacles in the room.
DAgger Finetuned
For difficult tasks, we refine human-trained policies with corrective robot demonstrations. Over two rounds of DAgger, our policies are able to complete long horizon, contact rich, and dexterous tasks fully autonomously.
Autonomous 1x
Fold Box
Pixel can learn precise, contact-rich bimanual manipulation with very few rounds of DAgger.
Autonomous 1x
Hang Jacket
Pixel can learn recovery and nuanced manipulation behaviors through DAgger to complete long horizon tasks.
More Experiments
Observation Space
Wrist Only
Wrist views are sufficient to grasp the mouse, but miss the broader context needed for whole-body positioning. Pixel misjudges the chair’s location and pulls it along while backing away.
Action Space
Joint Relative Actions
With joint relative actions, small errors accumulate during the reach. Pixel’s arm drifts against the box and pushes it aside instead of retrieving the canister.
Accumulating error in joint relative actions pulls the gripper out of alignment with the fridge door. The arm drifts upward and the door remains closed.
Joint relative actions allow the body posture to drift as the sequence progresses. After bringing the mouse to the pad, Pixel continues lifting its arm instead of settling at the end of the motion.
Environment Awareness
No Environment Awareness
Without scene geometry, matching the gripper target does not teach Pixel to lift its elbow above the box rim. The arm catches on the rim, preventing the gripper from reaching the canister.
Retargeting without environment awareness does not preserve clearance around the chair. Pixel’s arm collides with the chair back and stalls before grasping the mouse.
DAgger
Baseline
The human-trained policy performs the initial folds but struggles to tuck and secure the box’s side tabs. Repeated attempts at the same contact leave the box unfinished.
BibTeX
Publication details to be finalized.
@misc{pixel_umi,
title = {Pixel-UMI: Learning Environment-Aware Whole-Body Manipulation from Sensorized Human Demonstrations},
author = {Jason Jingzhou Liu and Tony Tao and Deepak Pathak and Ruslan Salakhutdinov},
year = {TBD}
}
