SyncWorld: Visual Calibration Turns World Models into Zero-Shot Simulators for Robots

Paper: SyncWorld: Visual Calibration Enables World Models as Zero-Shot Simulators Authors: Yuncong Yang, Zhengtao Han, Furkan Ozyurt, et al. arXiv: 2609.09155 Institutions: UMass Amherst, UC Berkeley, NYU, Harvard

The Problem: A Tower of Babel for World Models

World models predict the next visual frame given the current frame and an action, letting robots "imagine" outcomes before acting. But a fundamental obstacle exists: actions are not a universal language in pixel space.

• Change the camera position: the same "move forward 10 cm" produces different pixel displacements.
• Swap the robot arm: the same control signals yield different trajectories.
• Change the environment or robot base: visual consequences shift again.

Traditional solutions require collecting data and retraining for every new setup—impractical in real deployments.

Core Insight: Learn to Infer the Mapping, Not the Mapping Itself

SyncWorld's key idea: rather than learning a universal action-visual mapping, learn how to quickly infer the current environment's mapping from observation.

Visual Calibration

In a new environment, the robot performs a short calibration episode—simple probing actions demonstrating all controllable degrees of freedom (DoF)—while recording the corresponding visual changes. This is framed as in-context learning: like an LLM learning a new task from a few prompt examples, SyncWorld infers the setup-specific Action-Visual Mapping from a few paired action-visual examples.

This enables:

• Zero-shot generalization: no retraining in new environments, just a brief calibration
• Cross-environment transfer with a single model
• Cross-embodiment generalization across different robot arms

Training: Calibration Distillation

Training faces conflicting supervision: the same action values map to different visual changes across setups, so naive data mixing forces the model to fit incompatible mappings. SyncWorld's calibration distillation addresses this by:

1. Mixing training data from diverse setups (cameras, arms, backgrounds) 2. Systematically injecting calibration episodes as context during training 3. Teaching the model to interpret setup-specific action-visual pairs relative to the calibration context

Notably, SyncWorld also learns history-aware implicit calibration: when no explicit calibration episode exists, it can infer the mapping from prior interaction history—important since robots cannot recalibrate after every move.

Experimental Results

Tested across challenging benchmarks:

• Cross-camera viewpoints: training and test camera angles differ
• Cross-embodiment: different robot arms at test time
• Cross-environment: entirely different rooms/backgrounds

Findings:

• SyncWorld accurately simulates action outcomes in unseen environments, matching or exceeding prior state of the art, with standout cross-embodiment generalization
• Calibration quality matters: full demonstration of all 6 DoFs works best
• Even a few seconds of calibration suffices for effective adaptation

Test-Time Policy Improvement

The robot exploits the world model's imagination to search candidate actions "in its head," evaluate predicted outcomes, and execute the best one—no additional training required. The authors demonstrate this improves policy performance on real robots in new environments.

Why It Matters

Real-world robots face constantly shifting conditions: lighting changes, bumped cameras, product variations, post-maintenance recalibration. SyncWorld's zero-shot adaptation—seconds of automatic calibration instead of data collection and retraining—is a key step toward practical, deployable world models.

Future Directions

1. Online continual calibration: continuously refining environmental understanding during operation 2. Multimodal calibration: incorporating force and touch signals 3. Social calibration: calibrating by observing other robots 4. Meta-calibration learning: learning optimal calibration action sequences

Conclusion

SyncWorld shows that understanding a new environment doesn't require learning everything from scratch. A brief "eye contact"—demonstrating your actions and observing the world's response—is enough to bridge the gap, bringing general-purpose robotics closer to reality.

References

• Yang, Y., Han, Z., Ozyurt, F., et al. (2026). *SyncWorld: Visual Calibration Enables World Models as Zero-Shot Simulators*. arXiv:2609.09155.
• Hafner, D., et al. (2020). Dream to Control: Learning Behaviors by Latent Imagination. *ICML*.
• Ebert, F., et al. (2018). Visual Foresight: Model-Based Deep RL for Vision-Based Robotic Control. *CoRL*.
• Ye, S., et al. (2026). World Action Models are Zero-Shot Policies. arXiv:2602.15922.