Paper Overview
Field: Computer Vision / Robotics Authors: Hao Shi, Weiye Li, Bin Xie Published: 2025-06-06 arXiv: 2506.04876
Summary
Temporal modeling is essential for robotic manipulation: effective control requires both memory of past interactions and imagination of future states. However, most VLA (Vision-Language-Action) models rely primarily on the current observation and therefore struggle on long-horizon, temporally dependent tasks.
Cognitive science suggests humans rely on working memory to buffer short-lived context, the hippocampal system to preserve episodic memory of past experience, and internal models to imagine possible future state evolution. Inspired by these mechanisms, the authors propose MemoryVLA++, a full temporal modeling framework that equips VLA models with memory and imagination capabilities.
How It Works
1. Working memory: A pretrained VLM encodes the current observation into perceptual and cognitive tokens. 2. Episodic memory: These tokens query a perception-cognition memory bank that stores low-level details and high-level semantics of past interactions, updated through redundancy-aware consolidation. 3. Imagination: A world model imagines future states in a denoising latent space; the imagined latent variables are integrated into complete temporally aware tokens under memory guidance. 4. Action generation: These tokens condition a diffusion action expert to predict temporally consistent action sequences.
Results
Across 5 simulation benchmarks and 3 categories of real-robot tasks, the method achieves gains of:
- +9% on general manipulation
- +26% on memory-dependent tasks
- +28% on imagination-dependent tasks
- arXiv: https://arxiv.org/abs/2506.04876