InSight: When Robots Start Teaching Themselves New Skills
Paper: InSight: Self-Guided Skill Acquisition via Steerable VLAs Authors: Maggie Wang, Lars Osterberg, Stephen Tian arXiv: 2606.24884 Published: 2026-06-23
A Kitchen Metaphor
Imagine walking into an unfamiliar kitchen and being asked to cook a dish you have never made — say, omurice (Japanese omelette rice). You would probably:
1. Observe: eggs, rice, a spatula, a bowl — all familiar. 2. Decompose: beat the eggs, fry the rice, cook a thin omelette, wrap it up. 3. Combine: string these already-learned basic actions together into a new dish.
The beauty of this process is that nobody taught you how to make omurice step by step. You reused existing basic skills, plus a bit of observation and reasoning.
Now replace yourself with a robot. Traditionally, robots learn manipulation tasks through hundreds or thousands of human demonstrations. Want omurice? Demonstrate it 100 times. Want a different dish? Another 100 demonstrations. This is costly and fundamentally caps what the robot can ever learn.
InSight, from Stanford, proposes a different approach: let robots decompose existing skills, explore missing ones autonomously, and learn entirely new manipulation tasks — a framework that gives vision-language-action (VLA) models the ability to self-guide their skill acquisition.
The Core Problem: Why Robots Can't Learn New "Recipes"
Current VLA models (e.g., OpenVLA, RT-2) can learn impressive manipulation skills from demonstrations, but they live inside an invisible cage: their capabilities are bounded by the training data.
- Trained on "pick up the cup"? Done.
- Trained on "place it on the table"? Done.
- But "pick up the cup, fill it with water, then place it on the table" — a composition absent from the data — leaves the robot helpless.
- "Move the gripper above the bowl"
- "Lift upward"
- "Pour the water out of the bottle"
- Flipping a block (front to back)
- Closing a drawer
- Sweeping debris into a dustpan
- Unscrewing a threaded cap
- Pouring water from a bottle into a cup
- From specialized to general: Instead of a bespoke chef for every dish, InSight teaches basic cooking techniques and lets the robot invent new recipes.
- From passive to active: The robot doesn't wait to be fed data — it identifies what it is missing and goes to learn it. This proactivity is a hallmark of intelligence.
- From static to continual: The flywheel design makes capability growth a positive feedback loop — potentially an exponentially expanding one.
- Wang, M., Osterberg, L., Tian, S., et al. "InSight: Self-Guided Skill Acquisition via Steerable VLAs." arXiv:2606.24884, 2026.
- OpenVLA: An Open-Source Vision-Language-Action Model
- RT-2: Vision-Language-Action Models
- Socratic Models: Composing Zero-Shot Multimodal Reasoning
The deeper issue: these models are trained end-to-end as black boxes. We don't know how they internally combine "pick up" and "place," or whether they can recombine them into new skills. It's like a student memorizing 100 math answers without understanding arithmetic — the 101st variant leaves them stumped.
Human learning is different. From childhood we acquire a library of basic actions — grasp, push, pull, twist, pour, stack — and for any new task we:
1. Understand the goal 2. Decompose it into basic steps 3. Check which steps we already know 4. Learn the missing steps by trying 5. Combine and execute
The key properties are decomposability and composability. Skills aren't black boxes; they're Lego bricks.
InSight's Core Idea: Turning VLAs into Lego Bricks
InSight's goal in one sentence: make VLA models steerable at the level of "primitive action units," enabling autonomous skill acquisition.
"Steerable" is the key word. Instead of one end-to-end instruction like "put the cup on the table," the model accepts fine-grained commands such as:
These fine-grained commands are called primitives. Once a model is controllable at this granularity, it has a Lego kit it can recombine into structures it has never seen.
Stage 1: Automated Segmentation — Teaching the Robot Its Bricks
InSight first extracts primitives automatically from existing human demonstration data via an Automated Segmentation Pipeline, using two signals:
1. VLM-based step decomposition: A vision-language model (e.g., GPT-4V) watches a demonstration video and describes the steps — e.g., "move gripper above bowl → grasp bowl → lift → move above plate → tilt and pour → release." This gives each primitive a semantic label.
2. End-effector pose trajectories: The gripper's position, orientation, and open/close state over time provide precise temporal boundaries — for instance, a gripper closing marks the start of a grasp.
A useful analogy: segmenting a full orchestral recording into tracks. The VLM acts like a music critic ("this passage has violin, cello, piano..."), while the pose analysis is the waveform analysis that pinpoints exactly when each part begins and ends.
Stage 2: The VLM-Guided Data Flywheel
Once the robot has a library of primitives, how does it build new ones? Enter the VLM-Guided Data Flywheel — a loop that, once spinning, accelerates itself:
1. Identify missing skills: For a new task (e.g., "put the spoon from the drawer next to the bowl"), the VLM decomposes it into steps and InSight checks its skill library. Grasp spoon? Known. Move to bowl? Known. Open drawer? Missing!
2. Autonomous exploration: Rather than waiting for human teaching, the robot tries to learn the missing skill itself. The VLM proposes low-level control strategies ("move to the handle → grasp → pull back"), the robot executes them in simulation or the real world, and successful attempts are recorded as positive examples. Failures aren't wasted: the VLM analyzes why it failed and proposes a revised strategy (e.g., "press down slightly first to release the latch before pulling").
3. Automatic integration: Successful new skills are annotated, stored, and added to the VLA's training data. The robot can now reuse "open drawer" in any future task and compose it freely with existing skills for long-horizon tasks — no additional human demonstrations required.
A human analogy: a child learning to ride a bike already knows walking, running, balancing, and pedaling (their "primitives"). Riding requires combining them in a new way, so the child experiments — holding the wall, letting go a hand, falling a few times — until it clicks. And once learned, biking combines with everything else: riding to the store, dodging obstacles, carrying something in one hand.
Experimental Results: Simulation to the Real World
InSight was tested on skills never seen in human demonstrations:
Core findings:
1. Zero-demonstration skill acquisition: InSight learned all tested skills autonomously, without human demonstrations — something conventional VLAs cannot do. 2. Compositional generalization: Learned primitives compose into new long-horizon tasks (e.g., open drawer + grasp spoon + place beside bowl). 3. Continual learning: The flywheel compounds — each new skill enables more tasks, enabling more skills.
In real-world experiments, despite sensor noise, mechanical error, and environment variation, InSight completed most tasks — showing that segmentation and acquisition strategies learned in simulation transfer to the real world.
Why It Matters: Toward General Robot Intelligence
Limitations and Future Directions
1. Primitive granularity: Current primitives (e.g., "grasp") are relatively coarse; different shapes and materials may require finer control. 2. Dependence on the VLM: If the VLM mis-decomposes a task or proposes bad strategies, the whole flywheel fails. Robustness is a key open problem. 3. Cost of failed attempts: In the real world, exploration can be expensive (broken objects). Better simulation and safer learning strategies are needed.