MemoryVLA++: Teaching Robots Memory and Imagination to Fix the 'Goldfish Brain' Problem

*Note: This is a structured summary of a long-form Chinese forum post introducing the MemoryVLA++ architecture. Figures and claims below are as stated in the source post.*

The problem: VLA models have 'goldfish brain'

Vision-Language-Action (VLA) models such as OpenVLA and OpenVLA-OFT let robots see, understand language, and act—but they process only the current frame. A robot asked to fetch "the blue cup" may forget whether the instruction said blue or red, or where the cup was. The post reports that on the CALVIN simulation benchmark:

• Memory-dependent tasks (e.g., "pick up the cup you saw earlier"): existing VLA models succeed only ~30% of the time
• Imagination-dependent closed-loop tasks (e.g., "pour water without spilling"): success below 10%

Key points

• Biological inspiration: MemoryVLA++ mimics three human capabilities—working memory, long-term memory, and mental imagery for planning ahead.
• Memory encoder: A SigLIP-based vision-language encoder turns each timestep's image plus instruction into tokens, stored in a memory pool that acts as a priority queue, keeping important experiences and discarding irrelevant ones.
• Memory router: A lightweight network computes attention weights between the current state and all stored memory tokens, performing selective retrieval (like a librarian finding relevant books) rather than naive averaging.
• Mental imagination module: An autoregressive model (GPT-style token-by-token generation) produces imagined future action sequences. These imagined tokens are fed into the memory pool as "rehearsal" data, letting the robot plan before acting—like reading ahead in chess.
• Integration: Input to the Transformer becomes [language tokens] + [current image tokens] + [memory tokens...] + [imagination tokens...]. Memory and imagination are native attention inputs, not bolt-on modules.
• Memory management: temporal decay (recent memories weighted higher), importance filtering (low-information steps dropped), and compression of old low-priority memories into summaries (analogous to hippocampal consolidation).
• Imagination training: dual loss—prediction loss against ground-truth action/state sequences plus a task-success constraint so imagined paths are plausible plans, not random fantasies.

Key results

Memory-dependent tasks (CALVIN)

| Model | Memory tasks | General tasks | |---|---|---| | Base VLA | 29.3% | 45.7% | | + memory pool (no router) | 41.2% | 48.1% | | + memory router | 52.7% | 52.3% | | Full MemoryVLA++ | 61.8% | 56.9% |

Imagination-dependent tasks: 8.7% (base) → 23.4% (basic imagination) → 47.2% (trained imagination).

Real robots (UR5 and Franka Panda):

• Memory tasks: 34.5% → 68.3%
• Imagination tasks: 9.2% → 41.7%
• Long-horizon (10+ step) tasks: base VLA degrades to ~0%, MemoryVLA++ stays above 35%

Ablations: removing the memory router costs 18%; removing imagination costs 55% on imagination tasks; removing memory compression costs 22% on long-horizon tasks; removing temporal decay costs 15% on memory tasks.

Limitations and outlook

• The memory pool is finite; the imagination horizon is short; representations are token-level abstractions lacking rich sensory detail.
• Proposed future directions: hierarchical memory (working + episodic + semantic), sensory imagination (visual, tactile, auditory), and metacognition (knowing what it doesn't know).
• The post argues memory and imagination are not "advanced features" but foundations of intelligence: without memory, learning cannot occur; without imagination, planning cannot begin.

References cited in the post

1. *MemoryVLA++: Temporal Modeling via Memory and Imagination in Vision-Language-Action Models* — arXiv: 2606.09827 2. *CALVIN: A Benchmark for Language-Conditioned Policy Learning for Long-Horizon Robot Manipulation Tasks* — O. Mees et al., IEEE RA-L 2022 3. *OpenVLA: An Open-Source Vision-Language-Action Model* — M. Kim et al., arXiv: 2406.09246 4. *Attention Is All You Need* — A. Vaswani et al., NeurIPS 2017 5. *SigLIP: Sigmoid Loss for Language Image Pre-Training* — X. Zhai et al., arXiv: 2303.15343 6. *Neural Turing Machines* — A. Graves et al., arXiv:1410.5401 7. *The Feeling of What Happens* — Antonio Damasio, 1999