World Action Models Enable Continual Imitation Learning with Recurrent Generative Replays (REGEN)

Forum topic · 小凯 · 2026-06-27

Summary

This paper introduces REGEN (Recurrent Generative Replay), a continual imitation learning framework built on World Action Models (WAMs). Beyond predicting robot actions, WAMs can generate future visual observations, and REGEN leverages this generative capability to synthesize pseudo-replay trajectories. This allows a robot policy to rehearse previously learned tasks without storing original human demonstrations. During continual adaptation, REGEN recursively queries the WAM to synthesize pseudo-replays conditioned only on prior task instructions and current-task observations. Experiments in simulation and real-world manipulation show REGEN reduces catastrophic forgetting by up to 50% compared to sequential fine-tuning, approaching the performance of privileged experience replay methods that require access to real replay data. The authors also analyze limitations of generative replay, identifying long-horizon visual degradation and action-observation inconsistency as main bottlenecks. The results position WAMs as a promising foundation for continual robot learning without demonstration storage. Paper: arXiv 2606.27374.

Paper Overview

Field: Computer Vision, Robotics Authors: Manish Kumar Govind, Dominick Reilly, Smit Patel Published: 2026-06-27 arXiv: 2606.27374

Abstract

Going beyond predicting robot actions, World Action Models (WAMs) can also generate future visual observations. We build on this generative capability to propose Recurrent Generative Replay (REGEN), a continual imitation learning framework that synthesizes pseudo-replay trajectories, enabling a robot policy to rehearse previously learned tasks without storing their original human demonstrations. During continual adaptation, REGEN recursively queries the WAM to synthesize pseudo-replay trajectories conditioned only on prior task instructions and current-task observations. Experiments in both simulation and real-world manipulation settings show that REGEN reduces catastrophic forgetting by up to 50% relative to sequential fine-tuning, while approaching the performance of privileged experience replay methods that require access to real replay data. We analyze the factors limiting generative replay, identifying long-horizon visual degradation and action-observation inconsistency as primary bottlenecks. Our results establish WAMs as a promising foundation for continual robot learning without storing demonstrations.

Key Findings

WAMs extend beyond action prediction to generate future visual observations.
REGEN synthesizes pseudo-replay trajectories conditioned only on prior task instructions and current-task observations — no stored human demonstrations required.
Reduces catastrophic forgetting by up to 50% versus sequential fine-tuning in simulation and real-world manipulation.
Performance approaches privileged experience replay methods that retain real replay data.
Identified bottlenecks: long-horizon visual degradation and action-observation inconsistency.

--- *Auto-collected on 2026-06-27*