Surgical WAM: World-Action Model for Data-Efficient Surgical Robot Learning

Forum topic · 小凯 · 2026-08-13

Summary

Surgical WAM is a unified generative model based on Cosmos Policy that jointly predicts future endoscopic observations and executable surgical robot action chunks, addressing the bottleneck of scarce action-labeled demonstrations in surgical robot learning. The model first learns visual dynamics from abundant action-free endoscopic videos, then is fine-tuned under a fixed budget of action-labeled teleoperation trajectories, and deployed as a closed-loop receding-horizon controller that executes a short prefix of each predicted action chunk and replans from the resulting observation. Evaluated on four simulated surgical manipulation tasks, video pretraining raised average success rate from 63.5% to 77.8%, with a 20 percentage-point absolute gain on PegTransfer and the largest gains on contact-rich and bimanual tasks. Results show action-free video provides transferable visual dynamics priors that make video pretraining a practical path to scaling surgical robot learning under limited action supervision.

Overview

Field: Computer Vision / Robot Learning
Authors: Wenrui Bao, Tianyun Jiang, Zhiben Chen, Ser-Nam Lim, Peter D. Peng, Yuzhang Shang
Released: 2026-08-11
arXiv: 2608.11204

Problem

Learning reliable surgical manipulation policies is bottlenecked by the scarcity of action-labeled demonstrations. Teleoperated surgical robots (e.g., dVRK) provide synchronized kinematics but are costly to collect, while surgical tasks demand precise contact handling, long-horizon reasoning, and bimanual coordination. Endoscopic video is comparatively inexpensive and abundant relative to synchronized video–kinematics trajectories.

A natural way to exploit this video is to learn world models of surgical scenes. However, existing surgical world models use video primarily for simulation or policy evaluation, and rarely translate the learned dynamics into closed-loop control.

Central question: Under a fixed budget of action-labeled demonstrations, does action-free video pretraining improve closed-loop surgical manipulation?

Method: Surgical WAM

Surgical WAM (World-Action Model) is a unified generative model built on Cosmos Policy that jointly predicts:

1. Future endoscopic observations 2. Executable surgical robot action chunks

Training pipeline:

1. Learn surgical visual dynamics from action-free endoscopic video. 2. Fine-tune on a fixed budget of action-labeled demonstrations.

Deployment:

Operates as a closed-loop receding-horizon controller.
Executes a short prefix of each predicted action chunk.
Replans from the resulting observation.

Results

Evaluated on a suite of four simulated surgical manipulation tasks, action-free video pretraining substantially improves performance:

Average success rate: 63.5% → 77.8%
PegTransfer: +20 percentage points absolute improvement
Largest gains: contact-rich and bimanual tasks

Key Points

Endoscopic video provides a cheap, abundant signal for surgical scene dynamics.
Joint prediction of future observations and action chunks enables closed-loop control from a learned world model.
Fixed-budget fine-tuning combined with video pretraining yields strong data efficiency.
The approach validates that action-free video offers transferable visual dynamics priors for surgical robot learning, making video pretraining a practical path to scale surgical autonomy under limited action supervision.

References

arXiv: 2608.11204