World Action Models Enable Continual Imitation Learning with Recurrent Generative Replays (REGEN)

Forum topic · 小凯 · 2026-06-27

Summary

This paper introduces REGEN (Recurrent Generative Replay), a continual imitation learning framework built on World Action Models (WAMs), which generate not only robot actions but also future visual observations. Instead of storing original human demonstrations, REGEN synthesizes pseudo-replay trajectories by recursively querying the WAM, conditioned only on prior task instructions and current-task observations, allowing a robot policy to rehearse previously learned tasks during continual adaptation. Experiments in both simulation and real-world manipulation settings show that REGEN reduces catastrophic forgetting by up to 50% compared to sequential fine-tuning, while approaching the performance of privileged experience replay methods that require access to real replay data. The authors analyze limitations of generative replay, identifying long-horizon visual degradation and action-observation inconsistency as the main bottlenecks. The results position WAMs as a promising foundation for continual robot learning without stored demonstrations. Paper: arXiv 2606.27374.

Paper Overview

Fields: Computer Vision, Robotics Authors: Manish Kumar Govind, Dominick Reilly, Smit Patel Published: 2026-06-27 arXiv: 2606.27374

Abstract

Going beyond predicting robot actions, World Action Models (WAMs) can also generate future visual observations. The authors build on this generative capability to propose Recurrent Generative Replay (REGEN), a continual imitation learning framework that synthesizes pseudo-replay trajectories, enabling a robot policy to rehearse previously learned tasks without storing their original human demonstrations.

During continual adaptation, REGEN recursively queries the WAM to synthesize pseudo-replay trajectories conditioned only on prior task instructions and current-task observations.

Key Findings

REGEN reduces catastrophic forgetting by up to 50% relative to sequential fine-tuning in both simulation and real-world manipulation settings.
Its performance approaches that of privileged experience replay methods, which require access to real replay data.
The authors analyze factors limiting generative replay, identifying long-horizon visual degradation and action-observation inconsistency as the main bottlenecks.
The results establish WAMs as a promising foundation for continual robot learning without storing demonstrations.