Workspace Models: Lightweight Robotic Memory via Saliency-Driven VLM Distillation (arXiv 2609.20820)

Forum topic · 小凯 · 2026-09-19

Summary

This paper introduces "workspace tokens," a lightweight latent memory representation for long-horizon robotic manipulation. Conditioning policies on full histories can cause spurious correlations and degrade performance, while existing policy-memory methods rely on expensive VLM queries inside the inference loop to compress task-relevant information. The authors — Nitish Dashora, Douglas Chen, Idan Shenfeld, John Marangola, Pulkit Agrawal, and Max Simchowitz — invert this design: computationally intensive VLM queries happen only at training time. A VLM identifies the current and historical information needed to complete a task, and this information is distilled into the workspace token via a set-reconstruction decoder loss. At deployment, workspace tokens can directly substitute for observations, letting policies solve memory-intensive tasks without any VLM in the loop. Experiments in simulation and on hardware show that this representation is not only more computationally efficient but also yields better policy performance than history-conditioned or VLM-in-the-loop baselines. The paper is available on arXiv as 2609.20820.

Overview

Field: Machine Learning / Robotics Authors: Nitish Dashora, Douglas Chen, Idan Shenfeld, John Marangola, Pulkit Agrawal, Max Simchowitz Published: 2026-09-17 arXiv: 2609.20820

Abstract

Complex robotic manipulation tasks frequently require a long-term memory of past events and actions. As conditioning on full histories renders policies prone to spurious correlations and degrades performance, many approaches to policy memory involve compressing historical information through expensive VLM queries in-the-loop to process only task-salient information. This paper proposes an alternative approach in which computationally intensive VLM queries are made during training time to learn a lightweight latent memory that can be efficiently queried at deployment time.

The representation, called the workspace token, is trained by:

1. Using a VLM to identify current and historical information necessary for completing a task 2. Distilling these into the workspace token via a set-reconstruction decoder loss

In simulation and hardware experiments, the authors show that workspace tokens can directly substitute for observations at deployment time, enabling policies to solve memory-intensive tasks without a VLM inference loop. Interestingly, workspace tokens are not only more lightweight, but also lead to better policy performance.

Key Takeaways

Moves VLM computation from inference time to training time, cutting deployment costs
Workspace tokens serve as a compact, queryable latent memory replacing full observation histories
Demonstrated both in simulation and on real hardware
Reports improved policy performance alongside the efficiency gains

Paper: https://arxiv.org/abs/2609.20820