MemoryWAM: A Hybrid Memory Architecture for Robot Action Modeling

Forum topic · 小凯 · 2026-06-23

Summary

MemoryWAM is a world-action model that equips robots with human-like memory for long-horizon manipulation tasks. The paper addresses the memory-efficiency trade-off faced by existing approaches, where sliding-window methods forget critical long-range context and full-history caching explodes KV memory. Inspired by three-tier human memory, MemoryWAM combines a short-term sliding window (4 frames), anchor frames (initial 2 frames), and learnable gist tokens (8 per frame) within a Mixture-of-Transformers architecture. This hybrid design compresses the KV cache by 15x (from 120 visual tokens per frame to 8 gist tokens) while maintaining task success. On the RMBench simulation benchmark of 9 long-horizon bimanual tasks, MemoryWAM achieves 83.0% average success, outperforming LingBot-VA (78.2%) at lower latency. Ablations confirm gist tokens are essential: removing them drops Press Button success from 87% to 5%. Real-world ARX robot tests yield 90% on Shell Game and 75% on Look and Press, surpassing baselines.

Key points

Problem: Robots performing long-horizon manipulation tasks suffer from "amnesia." Sliding-window methods lose long-range context, while full-history KV caching causes memory blow-up and latency spikes. This is called the memory–efficiency trade-off.
Human-memory inspiration: The authors map three tiers of human memory to robot visual history — short-term memory (recent frames), event-boundary memory (anchor frames), and compressed long-term gist (gist tokens).
Architecture: MemoryWAM uses a Mixture-of-Transformers with a Video DiT (for dense feature extraction and memory cache maintenance) and an Action DiT (predicts next action chunk from cached representations). At inference, video generation is disabled to avoid expensive denoising.
Hybrid memory formula (paper Eq. 4): $C_{v \leq t}^v = C_v^{short} \cup C_v^{anchor} \cup C_v^{gist}$
Key innovation — Gist Token compression: Each frame contributes 8 learnable gist tokens instead of all 120 visual tokens, giving a 15× KV cache compression. KV complexity drops from $$O(NL)$$ to $$O(NM)$$ , where $$M = 8$$ . Later tokens attend only to gist tokens, not raw history.
Sim results (RMBench, 9 long-horizon bimanual tasks):
π₀.₅ (no memory): 10.4%
FastWAM (short window): 5.9%
LingBot-VA (full KV): 78.2%
MemoryWAM: 83.0%
Cover Blocks: 79% → 98% (+19%)
Real-world ARX robot:
Shell Game: 90% vs 65% (LingBot-VA)
Look and Press: 75% vs 70% (LingBot-VA)
Efficiency: Lowest latency at 1600 frames among Full Attention, TTT, and RNN baselines; memory scales as $$O(N/15)$$ .
Ablations (avg. success):
w/o Anchor: 74.0%
w/o Gist: 40.0% (Press Button collapses to 5%)
w/o Sliding Window: 82.5%
Full Attention: 91.5%
Hybrid (Ours): 92.5%
Insight: Selective, compressed memory is not a compromise — it beats full attention by filtering redundant context.
Limitations: Inherits video-diffusion reasoning limits; weak on multi-step symbolic logic. Future work points to dual-system neurosymbolic architectures.

Reference specs

| Parameter | Value | |---|---| | Total parameters | ~6B (Video 5B + Action 1B) | | Gist tokens / frame | 8 | | Visual tokens / frame | 120 | | Compression ratio | 15× | | Sliding window | 4 frames | | Anchor frames | 2 | | Action horizon | 16 steps | | Image resolution | 384×320 | | RMBench avg. success | 83.0% | | Real-world Shell Game | 90% |

Paper: *MemoryWAM: Efficient World Action Modeling with Persistent Memory* arXiv: 2606.20562v1 [cs.RO] Project page: https://yangsizhe.github.io/MemoryWAM/

MemoryWAM: A Hybrid Memory Architecture for Robot Action Modeling

Summary

Key points

Reference specs

Tags