Action Images: When Robots Learn to "Watch Movies" — Action Poetry in Pixel Space
> *"We see the world not only through our eyes, but through understanding what the eyes see."*
The Robot's Dilemma: Separation of Hand and Eye
When a human cuts a tomato in the kitchen, the eyes see a complete visual scene — a red tomato, a silver knife, the texture of the cutting board — and the body naturally coordinates. Traditional robot pipelines, however, capture camera images, extract features with computer vision algorithms, compress them into low-dimensional numeric representations, and have a controller compute motor commands. In this chain, vision and action are separated: the robot "sees" abstract numbers rather than a coherent visual world — like a chef deciding how to cut by reading a slip of paper saying "15cm long, 8cm diameter."
Action Images: Making Action Visible
In April 2026, researchers from Tsinghua University, MIT, and the Shanghai AI Laboratory proposed Action Images, built on a surprisingly simple idea:
> Instead of encoding actions as abstract numbers, why not turn the actions themselves into visual images?
How It Works
1. Multiview setup: multiple cameras observe the robot from different angles simultaneously. 2. Action encoding: the robot's 7-DoF action (3D end-effector position, 3D rotation, gripper open/close) is converted into visual trajectories. 3. Action video generation: a short video is generated showing the arm's motion over the next few seconds.
Key properties:
- Pixel-level grounding: actions are visible motion trajectories, not abstract vectors.
- Multiview consistency: actions seen from different angles agree, providing geometric constraints.
- Interpretability: humans can directly watch and judge whether planned motions are reasonable.
- Trajectory visualization: compute the arm's positions over the next T timesteps for each action.
- Optical-flow-style rendering: render these position sequences as optical-flow-like visualizations.
- Multiview projection: project the 3D trajectories onto each camera's 2D image plane.
- Action Images: 72% average success
- Baseline: 54% average success
- Compute cost: generating video is far more expensive than a low-dimensional vector; currently ~200ms per action.
- Action precision: pure pixel representation may be insufficient for sub-millimeter-precision tasks.
- Multi-robot coordination: still requires further research.
- Combining with large multimodal models (e.g., as the "brain") while Action Images serves as the "body's" expressive language.
- Human-robot collaboration interfaces: robots could show humans their planned motions as videos beforehand, and humans could correct them by editing the video.
- VR/AR applications: predicting user motions or generating virtual character motions.
- Haoyu Zhen, Zixian Gao, Qiao Sun, et al. "Action Images: End-to-End Policy Learning via Multiview Video Generation." arXiv preprint, April 2026.
- Ho, Jonathan, et al. "Imagen video: High definition video generation with diffusion models." arXiv 2022.
- Brooks, Tim, et al. "Video generation models as world simulators." OpenAI Research Blog, 2024.
- James, Stephen, et al. "RLBench: The robot learning benchmark & learning environment." IEEE Robotics and Automation Letters, 2020.
Why It Works
The key insight: modern video generation models already encode rich physical-world priors — how objects move, gravity, inertia, collisions, human and animal motion patterns, tool usage. Rather than retraining a dedicated control model, Action Images simply changes the input/output representation so pretrained video models can serve as policies.
Technical Details
Notably, no dedicated policy network is trained — the pretrained video diffusion model itself acts as the policy (zero-shot policy), and video, action, and observation all live in the same pixel space, enabling video-action joint generation, action-conditioned video generation, and action annotation.
Experimental Results
RLBench Simulation
| Metric | Action Images | Best Baseline | |---|---|---| | Average zero-shot success | 68.4% | 52.1% (+31% relative) | | Unseen object shapes | 61.2% | 38.7% | | Unseen camera angles | 55.8% | 29.3% | | 10% of training data | 51.3% | 24.6% |
Real World
On a real Franka robot performing grasping, stacking, and drawer-opening tasks:
Action Images also showed strong robustness to viewpoint changes, lighting changes, and object generalization.
Deeper Implications
1. From symbols to simulation: instead of describing the world precisely with symbols and numbers, the method directly "simulates" possible world changes in pixel space — closer to human cognition, where imagining "picking up a cup" is playing a mental video, not computing coordinates. 2. A new path for pretrained knowledge transfer: rather than modifying model architecture, modify input/output representations so video models' physics knowledge can be used directly for control. 3. An unexpected interpretability gain: you can see and debug robot trajectories directly. 4. A bridge to embodied intelligence: using pixels as a universal "interface language" between vision-language knowledge and physical action.
Limitations and Challenges
Future Directions
Conclusion
Teaching AI to "watch" actions rather than merely "compute" them improves robot performance and explores a new way of expressing intelligence. Perhaps one day, when a robot smoothly completes complex manipulation, we won't marvel at its math — we'll say: "It learned to see the world the way humans do."