Workspace Models: Lightweight Robotic Memory via Saliency-Driven Workspace Tokens

Forum topic · 小凯 · 2026-09-19

Summary

This arXiv paper (2609.20820) by Nitish Dashora, Douglas Chen, Idan Shenfeld, John Marangola, Pulkit Agrawal, and Max Simchowitz introduces workspace tokens, a lightweight latent memory representation for robotic manipulation policies. Complex manipulation tasks require long-term memory of past events and actions, but conditioning policies on full histories invites spurious correlations and degrades performance. Existing methods compress history via expensive vision-language model (VLM) queries in the inference loop. The proposed alternative performs computationally intensive VLM queries during training: a VLM identifies current and historical information necessary for task completion, and this saliency is distilled into a workspace token through a set-reconstruction decoder loss. At deployment, workspace tokens can directly substitute for observations, enabling policies to solve memory-intensive tasks without a VLM in the inference loop. Experiments in simulation and on hardware show this approach is not only more lightweight but also improves policy performance. The post is an auto-collected paper summary from the zhichai.net forum.

Paper Overview

Field: Machine Learning (ML)
Authors: Nitish Dashora, Douglas Chen, Idan Shenfeld, John Marangola, Pulkit Agrawal, Max Simchowitz
arXiv: 2609.20820

Abstract

Complex robotic manipulation tasks frequently require a long-term memory of past events and actions. As conditioning on full histories renders policies prone to spurious correlations and degrades performance, many approaches to policy memory involve compressing historical information through expensive VLM queries in-the-loop to process only task-salient information.

This paper proposes an alternative approach in which computationally intensive VLM queries are made during train-time to learn a lightweight latent memory that can be efficiently queried at deployment time. The representation, called the workspace token, is trained by:

1. Using a VLM to identify current and historical information necessary for completing a task 2. Distilling this information into the workspace token via a set-reconstruction decoder loss

In simulation and hardware experiments, workspace tokens can directly substitute for observations at deployment time, allowing policies to solve memory-intensive tasks without a VLM in-the-loop. Interestingly, the authors find that workspace tokens are not only more lightweight, but also yield better policy performance.

Key Takeaways

Moves expensive VLM saliency queries from inference time to training time
Produces a lightweight latent memory (workspace token) queryable at deployment
Distillation via set-reconstruction decoder loss preserves task-relevant information
Validated in both simulation and real hardware experiments, with improved performance over in-the-loop VLM memory baselines

---

*Auto-collected on 2026-09-19*