MemoryVLA++: Temporal Modeling via Memory and Imagination for Vision-Language-Action Models in Robotic Manipulation

Forum topic · 小凯 · 2026-06-10

Summary

MemoryVLA++ (arXiv:2506.04876) is a vision-language-action framework that adds temporal modeling—memory and imagination—to robotic manipulation. Inspired by human cognition, it uses a pretrained VLM to encode observations into perceptual and cognitive tokens forming working memory, queries a perceptual-cognitive memory bank storing low-level details and high-level semantics of past interactions (updated via redundancy-aware consolidation), and employs a world model to imagine future states in a denoised latent space. The resulting temporally aware tokens condition a diffusion action expert to predict temporally consistent action sequences. Across 5 simulation benchmarks and 3 real robot task categories, it improves general manipulation by +9%, memory-dependent tasks by +26%, and imagination-dependent tasks by +28% over baselines.

Paper Overview

Research area: Computer Vision / Robotics
Authors: Hao Shi, Weiye Li, Bin Xie
Published: 2025-06-06
arXiv: 2506.04876

Abstract

Temporal modeling is essential for robotic manipulation, as effective control requires both memory of past interactions and imagination of future states. However, most VLA models rely primarily on the current observation and therefore struggle with long-horizon, temporally dependent tasks.

Cognitive science suggests that humans rely on working memory to buffer short-lived context, the hippocampal system to preserve episodic memory of past experience, and internal models to imagine possible future state evolution. Inspired by these mechanisms, the authors propose MemoryVLA++, a full temporal modeling framework that equips VLA models with memory and imagination capabilities for robotic manipulation.

How It Works

1. Working memory: A pretrained VLM encodes the current observation into perceptual and cognitive tokens. 2. Episodic memory bank: These tokens query a perceptual-cognitive memory bank to retrieve relevant historical context. The bank stores both low-level details and high-level semantics of past interactions, updated through redundancy-aware consolidation. 3. Imagination: A world model imagines future states in a denoised latent space; the imagined latent variables are integrated with memory guidance into complete temporally-aware tokens. 4. Action generation: These tokens condition a diffusion action expert to predict temporally consistent action sequences.

Results

Across 5 simulation benchmarks and 3 categories of real robot tasks, MemoryVLA++ achieves improvements of:

+9% on general manipulation tasks
+26% on memory-dependent tasks
+28% on imagination-dependent tasks

These results demonstrate that combining cognitive-inspired memory retrieval with latent-space imagination substantially improves long-horizon robotic control compared to observation-only VLA baselines.