SIM1: A Physics-Aligned Simulator That Scales Deformable Object Manipulation Data via Real-to-Sim-to-Real

Forum topic · 小凯 · 2026-04-12

Summary

SIM1 (arXiv:2504.07774) is a real-to-sim-to-real data engine for robot learning with deformable objects such as cloth, ropes, and soft items. Instead of relying on approximate simulators and domain randomization, SIM1 builds metric-consistent digital twins from a small number of real demonstrations, calibrates deformable dynamics through elastic modeling (bending, stretching, shearing stiffness, damping), and then uses diffusion models to generate diverse manipulation trajectories with quality filtering. Experiments on cloth folding, fabric flattening, knot handling, and soft-object grasping show that policies trained purely on synthetic data match real-data baselines at a 1:15 data equivalence ratio, achieve about 90% zero-shot success on unseen real scenes, and improve generalization by roughly 50% over traditional sim-to-real pipelines. The key insight is that for physical interaction tasks, physics consistency matters more than visual fidelity: aligning simulator dynamics with reality eliminates the sim-to-real gap rather than papering over it. The post also discusses extensions to fluids, granular materials, articulated objects, and fully automated scene reconstruction as future work.

SIM1: Feeding Real Robots with Virtual Data in Deformable Worlds

Paper: SIM1: Physics-Aligned Simulator as Zero-Shot Data Scaler in Deformable Worlds arXiv: 2504.07774 Field: Robot learning / Sim-to-real transfer

Why deformable objects are hard

Rigid-body manipulation has 6 degrees of freedom per object; deformable objects (clothing, cloth, ropes, food) have effectively infinite degrees of freedom — they fold, tangle, and knot. Data collection suffers from:

Hard annotation: no clear "correct" state for cloth
Expensive collection: manual environment resets, slow real-world trials
Limited diversity: narrow material and lighting coverage

Traditional sim-to-real fails here because simulators are ungrounded: geometry looks wrong, soft-body dynamics are fragile to parameter drift, and rigid-body action primitives transfer poorly.

The SIM1 approach: real-to-sim-to-real

1. Scene digitization: from a few real demonstrations, reconstruct a metric-consistent digital twin (geometry, physics parameters, trajectories). 2. Physics calibration: align deformable dynamics via elastic modeling — bending/stretching/shearing stiffness and damping parameters. 3. Data generation & filtering: diffusion models generate diverse trajectories; quality filtering and domain randomization scale dozens of real demos into thousands of high-quality synthetic samples.

The core philosophy: don't make reality fit the simulation — make the simulation fit reality (physics-aligned, not merely approximate).

Results

1:15 data equivalence: policies trained on pure synthetic data match real-data baselines (≈1500 synthetic samples ≈ 100 real samples)
~90% zero-shot success rate when deployed directly on unseen real scenes
~50% generalization improvement over traditional sim-to-real baselines on new objects and layouts

Why it works

For physical interaction tasks, physics consistency matters more than visual fidelity. A simulator with simple visuals but correct dynamics transfers far better than a photorealistic one where cloth behaves like plastic film. Alignment eliminates the sim-to-real gap at the root instead of asking the network to learn around it.

Outlook

Extensions to fluids, granular materials, articulated objects, and human-robot interaction
Fully automated scene reconstruction and physics parameter estimation from video
Synergy with real data: cold-start, data augmentation, and continual learning from deployed feedback

Reference: Zhou, Y., Liu, H., Jiang, X., et al. "SIM1: Physics-Aligned Simulator as Zero-Shot Data Scaler in Deformable Worlds." arXiv:2504.07774.