Action Images: Teaching Robots to 'Watch' Actions as Video with Multiview Video Generation

Forum topic · 小凯 · 2026-04-08

Summary

Researchers from Tsinghua University, MIT, and the Shanghai AI Laboratory propose Action Images, a method that represents robot actions as multiview videos instead of abstract low-dimensional vectors. By rendering 7-DoF robot trajectories as visual motion patterns projected onto multiple camera views, the approach lets pretrained video diffusion models act directly as policies, reusing their learned physical priors about motion, gravity, and object interaction. In RLBench simulations, Action Images achieved an average zero-shot success rate of 68.4% versus 52.1% for the best baseline, with stronger generalization to unseen objects (61.2% vs 38.7%) and camera angles (55.8% vs 29.3%), and better data efficiency. On a real Franka robot performing grasping, stacking, and drawer-opening tasks, it reached 72% average success versus 54% for baselines. The method also improves interpretability, since planned motions are directly visible as video. Remaining challenges include the ~200ms inference latency per action, precision limits for sub-millimeter tasks, and multi-robot coordination. The work suggests a path toward embodied AI by changing input/output representations rather than model architecture.

Action Images: Teaching Robots to 'Watch' Actions as Video

> *"We not only see the world through our eyes, but understand our eyes through comprehension."* — attributed to Aristotle

The Robot's Dilemma: Separated Hand and Eye

When a human chops a tomato, they simply look at the tomato, the knife, and the cutting board, and act. Traditional robot pipelines break this into stages: capture images, extract features with computer vision, compress them into low-dimensional vectors, then compute motor commands from those numbers.

The core problem: vision and action are separated. The robot "sees" abstract numbers rather than a coherent visual world — like a chef deciding how to cut by reading a note saying "15cm long, 8cm diameter" instead of looking at the ingredient.

What Are Action Images?

In April 2026, researchers from Tsinghua University, MIT, and the Shanghai AI Laboratory proposed Action Images, built on a strikingly simple idea:

> Instead of encoding actions as abstract numbers, turn the actions themselves into visual images.

The method works as follows:

1. Multiview setup: Multiple cameras observe the robot from different angles. 2. Action encoding: The robot's 7-DoF action (3D end-effector position, 3D rotation, gripper state) is converted into visual trajectories. 3. Action video generation: A short video is generated showing the robot arm's motion over the next few timesteps.

Key properties:

Pixel-level grounding: Actions appear as real, visible motion trajectories.
Multiview consistency: The same action is consistent across camera views, providing geometric constraints.
Interpretability: Humans can directly watch and judge whether planned motions are reasonable.

Why It Works

Modern video generation models have absorbed rich physical priors from massive video data — how objects move, how gravity, inertia, and collisions behave, how humans and tools are used. The core insight: rather than training a new robot-control model from scratch, reuse pretrained video models by changing only the input/output representation.

Technical Details

Representing 7-DoF actions: For each action, the system computes the robot arm's position sequence over future timesteps, renders it in an optical-flow-like style, and projects the 3D trajectory onto each camera's 2D image plane.
Fusion with video diffusion models: No dedicated policy network is trained — the pretrained video diffusion model itself acts as the policy. This enables zero-shot policies, leverages pretraining knowledge, and unifies video, action, and observation in one pixel space (supporting video-action joint generation, action-conditioned video generation, and action annotation).

Experimental Results

RLBench Simulation

68.4%

61.2%

55.8%

51.3%

Real-World Evaluation

Action Images: 72% average success
Baseline: 54%

Action Images also showed strong robustness to viewpoint changes, lighting changes, and object generalization.

Deeper Implications

1. From symbols to simulation: Instead of describing the world with symbols and coordinates, actions are "simulated" directly in pixel space — closer to how humans mentally replay picking up a cup. 2. A new path for knowledge transfer: Change the representation, not the model architecture, to unlock what video models already know about physics. 3. Interpretability as a bonus: Planned trajectories can be watched, judged, and debugged visually — valuable for real-world robotics. 4. A bridge to embodied intelligence: Pixel space serves as a universal interface between vision-language knowledge and physical action.

Limitations and Challenges

Compute cost: Generating video is heavier than generating a low-dimensional vector; an action currently takes ~200ms.
Action precision: Pure pixel representations may not suffice for sub-millimeter-precision tasks.
Multi-robot coordination: Extending to multi-robot settings requires further research.

Future Outlook

Combining with large multimodal models: Models like GPT-4V or Gemini could serve as the "brain," with Action Images as the "body's" expressive language.
Human-robot collaboration: Generated action videos could preview a robot's plan, letting humans correct behavior by editing the video.
VR/AR applications: Predicting user motions or generating virtual character movements.

Conclusion

Turning actions into visible video seems simple but touches a deep question: what is the essence of intelligence? When we teach AI to *watch* actions rather than merely compute them, we both improve robot performance and explore a new form of expressing intelligence.

References

Haoyu Zhen, Zixian Gao, Qiao Sun, et al. "Action Images: End-to-End Policy Learning via Multiview Video Generation." arXiv preprint, April 2026.
Ho, Jonathan, et al. "Imagen video: High definition video generation with diffusion models." arXiv 2022.
Brooks, Tim, et al. "Video generation models as world simulators." OpenAI Research Blog, 2024.
James, Stephen, et al. "RLBench: The robot learning benchmark & learning environment." IEEE Robotics and Automation Letters, 2020.

*This article was written using the Feynman technique with everyday analogies; for any discrepancies, defer to the original paper.*