InSight: When Robots Start 'Learning to Cook' on Their Own
Paper: InSight: Self-Guided Skill Acquisition via Steerable VLAs Authors: Maggie Wang, Lars Osterberg, Stephen Tian arXiv: 2606.24884 Published: 2026-06-23
🍳 A Kitchen Metaphor
Imagine walking into an unfamiliar kitchen and being asked to make a dish you've never tried—say, omurice. You would observe the ingredients, decompose the task into basic actions you already know (cracking eggs, frying rice, making an omelet, wrapping), then compose them in sequence. Crucially, nobody hands you a step-by-step demonstration—you combine existing skills with a bit of observation and reasoning.
Traditionally, robots learn manipulation tasks the hard way: hundreds of human demonstrations per task, making it impossible to learn skills outside the training data. A new Stanford paper proposes a different approach: let robots decompose existing skills and autonomously explore missing ones. That is InSight—a framework that gives vision-language-action (VLA) models the ability to learn new manipulation tasks on their own.
🧩 Why Current Robots Can't Learn a 'New Dish'
Existing VLA models (e.g., OpenVLA, RT-2) can imitate demonstrations, but their capabilities are trapped by the boundaries of their training data. Ask for a novel composition of known actions—"pick up the cup, fill it with water, then place it on the table"—and the robot often fails.
The deeper problem: these models are trained end-to-end as black boxes. We don't know how they combine "pick up" and "place down" internally, or whether they can recombine "pick up" and "pour" into a new skill. It's like a student memorizing 100 math answers without understanding arithmetic.
Human learning works differently: we build a repertoire of primitives (grasp, push, pull, twist, pour, stack), then for any new task we (1) understand the goal, (2) decompose it into steps, (3) check which steps we already know, (4) learn the missing ones, and (5) execute the composition. The key properties are decomposability and composability.
🔧 Core Idea: Turning VLAs into 'Lego Bricks'
InSight's goal in one sentence: make VLA models steerable at the level of primitive action units, so they can autonomously acquire new skills. Instead of mapping a single instruction to a whole action trajectory, a steerable model accepts fine-grained commands such as:
- "Move the gripper above the bowl"
- "Lift up"
- "Pour the water out of the bottle"
- Flipping a block from front to back
- Closing a drawer
- Sweeping debris into a dustpan
- Unscrewing a threaded bottle cap
- Pouring water from a bottle into a cup
- From specialized to general: Instead of task-specific data collection, InSight builds a general repertoire of primitives that can be recombined for any new task.
- From passive to active: The robot actively identifies what it lacks and tries to learn it—a hallmark of intelligence.
- From static to continual: Each new task can yield new skills, creating an accelerating intelligence flywheel.
- Wang, M., Osterberg, L., Tian, S., et al. "InSight: Self-Guided Skill Acquisition via Steerable VLAs." arXiv:2606.24884, 2026.
- OpenVLA: An Open-Source Vision-Language-Action Model
- RT-2: Vision-Language-Action Models
- Socratic Models: Composing Zero-Shot Multimodal Reasoning
Once a model can be controlled at this granularity, it possesses a set of Lego bricks that can be assembled into structures it has never seen.
🔄 Stage 1: Automated Segmentation—Teaching the Robot Its 'Bricks'
InSight first extracts primitives from existing human demonstration data via an Automated Segmentation Pipeline, using two signals:
1. VLM planning decomposition: A powerful VLM watches a demonstration video and describes the steps (e.g., move gripper above the bowl → grasp → lift → move over the plate → tilt and pour → release). This gives each segment a semantic label. 2. End-effector pose: The gripper's position, orientation, and open/close state over time provide precise temporal boundaries. For example, a transition from open to closed likely marks the start of a grasp.
A helpful analogy: it's like separating a symphony recording into tracks. The VLM is the music critic telling you which instruments are playing; the end-effector pose analysis is the waveform analysis pinpointing exactly when each instrument plays.
The result: from one end-to-end demonstration, automatically extract semantically labeled, time-bounded primitive action units.
🎡 Stage 2: The Data Flywheel—Letting Robots Explore New Skills
With a library of primitives, InSight runs a VLM-Guided Data Flywheel:
1. Identify missing skills. Given a new task (e.g., "take the spoon from the drawer and place it next to the bowl"), the VLM decomposes it into steps and InSight checks them against its skill library. Grasp spoon: known. Move to bowl: known. Open drawer: missing! 2. Attempt autonomously. The VLM proposes low-level control strategies ("move to the drawer handle," "grasp," "pull back"), which the robot executes in simulation or the real world. Successes are recorded as positive examples. On failure, the VLM analyzes the cause and proposes a revised strategy (e.g., "press down first to release the latch before pulling"). 3. Integrate automatically. Successful attempts are labeled, stored, and added to the VLA's training set. New skills become callable—and freely combinable with existing ones—enabling longer-horizon tasks without further human demonstrations.
Think of a child learning to ride a bike: the primitives (walking, balance, pedaling) already exist; the child identifies the missing composition, explores, falls a few times, and eventually masters it—then reuses that skill everywhere.
🧪 Experimental Results
InSight was tested on skills never seen in human demonstrations:
Key findings
1. Zero-demonstration skill acquisition: InSight learned all tested skills without human demonstrations—something conventional VLAs cannot do. 2. Compositional generalization: Learned primitives compose into new long-horizon tasks (e.g., "open drawer + grasp spoon + place beside bowl"). 3. Continual learning: The data flywheel keeps expanding the skill library, enabling increasingly complex tasks in a positive feedback loop.
Despite real-world challenges (sensor noise, mechanical error, environment variation), InSight completed most real-robot tasks, showing that skill segmentation and acquisition strategies learned in simulation transfer to reality.
🌌 The Bigger Picture
🎯 Limitations and Future Work
1. Primitive granularity: Current primitives are relatively coarse; manipulating objects of different shapes and materials may need finer-grained decomposition. 2. VLM dependence: Task decomposition and strategy proposals rely heavily on VLMs (e.g., GPT-4V); incorrect decompositions can break the flywheel. Robustness is a key open problem. 3. Cost of failed attempts: Real-world exploration can be expensive (e.g., breaking objects). Better simulation and safer learning strategies are needed.