Overview
Field: Computer Vision Authors: Wenrui Bao, Tianyun Jiang, Zhiben Chen, Ser-Nam Lim, Peter D. Peng, Yuzhang Shang Published: 2026-08-11 arXiv: 2608.11204
Motivation
Learning reliable surgical manipulation policies is bottlenecked by the scarcity of action-labeled demonstrations: teleoperated surgical robot (e.g., dVRK) trajectories with synchronized kinematics are costly to collect, while surgical tasks demand precise contact handling, long-horizon reasoning, and bimanual coordination.
Endoscopic video is comparatively inexpensive and abundant relative to synchronized video–kinematics trajectories, and a natural way to exploit it is to learn world models of surgical scenes. However, existing surgical world models use video primarily for simulation or policy evaluation, and rarely translate the learned dynamics into closed-loop control.
This gap raises the paper's central question: under a fixed budget of action-labeled demonstrations, does action-free video pretraining improve closed-loop surgical manipulation?
Method
- Surgical WAM (Surgical World-Action Model) is a unified generative model based on Cosmos Policy that jointly predicts future endoscopic observations and executable surgical robot action chunks.
- It first learns surgical visual dynamics from action-free video, then fine-tunes on a fixed action-labeled demonstration budget.
- At deployment, it acts as a closed-loop receding-horizon controller: it executes the short prefix of each predicted action chunk and replans from the resulting observations.
- On four simulated surgical manipulation task suites, video pretraining improves average success rate from 63.5% to 77.8%.
- PegTransfer gains an absolute 20 percentage points.
- The largest improvements appear on contact-rich and bimanual tasks.
Results
Takeaway
Action-free video provides transferable visual dynamics priors for surgical robot control learning under limited action supervision, making data-efficient video pretraining a practical path for scaling surgical robot learning.
--- *Auto-collected on 2026-08-13*