Summary
InSight is a framework enabling vision-language-action (VLA) models to autonomously acquire new manipulation skills beyond their training data by making them steerable at the primitive-action level (e.g., 'move gripper to the bowl', 'lift upward', 'pour the bottle'). The system has two stages: (1) an automated segmentation pipeline that partitions demonstrations into labeled primitives using VLM plan decomposition and end-effector poses, and (2) a VLM-guided data flywheel that identifies missing primitives for novel tasks, autonomously attempts demonstrations using VLM-proposed low-level control, and automatically labels and integrates successful demos into VLA training. Evaluated in simulation and real-world tasks including flipping a block, closing a drawer, sweeping, twisting, and pouring, InSight acquires target skills without human demonstrations, and learned primitives can be composed for new long-horizon tasks. Paper: arXiv 2506.14748, by Maggie Wang, Lars Osterberg, and Stephen Tian.
Paper Overview
Field: ML
Authors: Maggie Wang, Lars Osterberg, Stephen Tian
Posted: 2026-06-24
arXiv: 2506.14748
Summary
Vision-language-action (VLA) models can learn manipulation skills from demonstrations, but their capabilities are bounded by the skills present in the training data. This post introduces InSight, a framework that unlocks autonomous skill acquisition by making VLAs steerable at the primitive-action level (e.g., "move gripper to the bowl", "lift upward", "pour the bottle").
InSight consists of two primary stages:
1. Automated segmentation pipeline — partitions demonstrations into labeled primitives via VLM plan decomposition and end-effector poses, enabling VLA primitive steerability.
2. VLM-guided data flywheel — identifies missing primitives required to accomplish a novel task, autonomously attempts demonstrations of the missing primitives using VLM-proposed low-level control, and automatically labels, stores, and integrates successful demonstrations into the VLA training set.
Evaluation
InSight was evaluated on simulation and real-world manipulation tasks, including:
- Flipping a block
- Closing a drawer
- Sweeping
- Twisting
- Pouring
None of these target skills required human demonstrations. Once learned, the primitives can be composed to execute new long-horizon tasks without additional human demonstrations.
Conclusion
The findings suggest that primitive-action steerability provides a practical foundation for continual skill acquisition in VLA policies.
---
*Auto-collected 2026-06-25*
This page is an English static mirror generated for search and AI citation.
It may be a full translation or structured summary of the Chinese original.
Canonical interactive discussion lives on the Chinese page:
https://zhichai.net/topic/178208097