Learning from Three Masters: A Humanoid Robot's Training Journey (HANDOFF Paper)
The Problem: The Command-Space Dilemma in Humanoid Control
Imagine a Unitree G1 humanoid robot — 1.6 meters tall with 23 actuated joints. It can walk, turn, squat, and do simple gymnastics. But the real challenge is performing tasks: executing complex manipulation based on natural-language instructions like "put the red block on the table into the blue box."
The core difficulty lies in the command space — the interface between task planning and motor execution:
- Task-level commands ("put the cup in the dishwasher"): intuitive, but too abstract for a controller to execute directly.
- Motor-level commands ("motor 3 rotates 15 degrees, motor 7 rotates -8 degrees..."): precise, but impossible for planners or humans to specify.
- Motion-level commands ("move right hand to (0.5, 0.3, 0.2), palm down..."): a middle ground, but still requires exact coordinates that semantic planners cannot easily generate.
- Whole-body motion tracker: masters human-like motion via motion-capture data, but cares about style, not task success.
- Locomotion specialist: handles stable walking, balance, and terrain adaptation, but knows nothing about upper-body manipulation.
- Fall-recovery expert: handles disturbance rejection and safe falls, but is only relevant in dangerous situations.
- Body state (joint angles, velocities, center of mass)
- Task description (natural language)
- Environment information (terrain, obstacles, contacts)
- Velocity tracking: state-of-the-art performance — accurate speed tracking with balance, achieved by fusing locomotion and recovery knowledge.
- Robust manipulation workspace: HANDOFF achieves one of the largest robust manipulation workspaces — grasping from varied angles and distances, manipulating while walking, adapting to different object shapes and weights.
- Language-driven real-world tasks: combined with a VLM planner, HANDOFF executes instructions like "put the red block into the blue box": the VLM decomposes the task into subgoals (approach, detect, grasp, move, place), each translated into HANDOFF's command space.
- HANDOFF + VLM planner + Unitree G1 forms a prototype of a general task-execution platform: language in, vision understanding, coordinated whole-body out, no task-specific training.
- Robust whole-body control opens the door to robotic expression — gestures, dance, sports.
- It lays technical groundwork for physical human-robot collaboration: carrying objects together, dancing in sync, playing sports.
- Yang, L., Li, J., Poddar, N., et al. (2026). *HANDOFF: Humanoid Agentic Task-Space Whole-Body Control via Distilled Complementary Teachers*. arXiv: cs.RO.
- Related concepts: Humanoid Robot, Whole-Body Control, Multi-Teacher Distillation, KL Distillation, Mixture of Experts, Gating Network, Unitree G1, VLM, Task Planning
There is a mismatch between what task planners (LLMs/VLMs) can output — semantic instructions — and what whole-body controllers need as input. HANDOFF bridges this gap.
HANDOFF's Command Space Design
HANDOFF defines an interface that is intuitive, general, modular, and expressive. It includes:
1. Task semantics — natural-language task descriptions 2. Target poses — end-effector positions and orientations 3. Contact constraints — which body parts may touch the environment 4. Speed/force preferences — how fast and how forcefully to act 5. Safety boundaries — forbidden regions and actions
Think of it as a standardized blueprint: the architect (task planner) draws designs without knowing construction details; the construction crew (controller) translates the blueprint into concrete steps.
Three Complementary Expert Teachers
HANDOFF uses multi-teacher distillation, training one student policy from three specialists:
These teachers are complementary — covering elegance, stability, and safety — but their advice can conflict. A motion tracker might demand a high-difficulty pose the locomotion expert deems unstable.
Distillation Alchemy: KL Loss + Context-Conditioned Gating
KL Distillation
For each training sample, all three teachers produce action distributions. The student's loss is a weighted sum of KL divergences from each teacher — the student synthesizes all three opinions rather than following a single one.
Gating Network
Naive averaging is suboptimal when one teacher is clearly more reliable in a given situation. HANDOFF introduces a context-conditioned gating network that takes:
and outputs weights w1, w2, w3 (summing to 1) for the three teachers. Like a smart study advisor, it emphasizes the locomotion teacher while walking, the motion tracker during manipulation, and the recovery expert when pushed.
Results: The Apprentice's Final Exam
Crucially, this required no task-specific data and no controller fine-tuning — HANDOFF is a general-purpose whole-body controller; any task a VLM planner can decompose, HANDOFF can execute.
Deeper Philosophy: Whole-Body Intelligence and Divide-and-Conquer
1. Wholeness: Unlike fixed-base arms, humanoids are whole-body systems — walking and manipulation cannot be separated. HANDOFF coordinates both simultaneously rather than "move first, then operate."
2. Divide, then fuse: Train specialist teachers separately, then unify them via distillation and gating — mirroring how humans learn disciplines separately and combine them in practice.
3. The value of complementarity: Fusing experts with overlapping skills adds little; fusing *complementary* experts covering different scenario needs yields a system more capable than any single teacher. This suggests building multiple complementary specialists — and learning to fuse them — rather than chasing one omniscient model.