In-Context Robot Learning with VLM Agents: Introducing GPT-Policy

Forum topic · 小凯 · 2026-09-18

Summary

A new paper (arXiv:2609.19138) introduces GPT-Policy, a general agentic framework for in-context robot learning built on commercial vision-language models (VLMs) such as GPT-6 Astra. The framework addresses a core challenge in embodied AI: since no finite set of demonstrations can cover every task a robot will encounter, robots must learn from context at deployment time—without gradient updates or persistent changes to task parameters. GPT-Policy integrates three components: a context compiler that preserves task-relevant visual transformations, a VLM that proposes robot-tool actions, and a constraint controller that validates and executes each action while reporting outcomes. Real-robot experiments evaluate task success rates, efficiency, cross-model comparisons, and controlled context ablations. Results show that human video demonstrations improve task completion even without robot action labels, and aligned action references further boost performance on contact-sensitive tasks. The paper positions GPT-Policy as a step toward in-context robotic adaptation and provides an empirical basis for translating general VLM capabilities into physical robot behavior.

This post introduces the paper "In-Context Robot Learning with VLM Agents" (arXiv: 2609.19138), filed under computer vision, authored by Dongzhou Cheng, Taoran Yi, Ye Fang, Xingwu Zhang, Fan Feng, Yixuan Li, Gengxiong Zhuang, Rongze Wang, Shuai Yang, Wei Song, Weizhi Xue, Minyan Wu, Jie Gui, Jiaqi Wang, and Tong Wu.

The Problem

Enabling robots to adapt to unfamiliar environments as readily as humans remains a moonshot goal of embodied AI. No finite collection of demonstrations can cover every task and situation a robot will encounter, making the ability to learn from context at deployment essential for generalization. Such in-context learning (ICL), however, remains largely beyond the reach of existing robotic policies.

The broad agentic capabilities of commercial vision-language models (VLMs), such as GPT-6 Astra, raise a compelling question: can these models learn from demonstrations, examples, and interaction feedback, then translate that information into executable and verifiable robot behavior from a new initial state—without gradient updates or persistent changes to task-specific parameters?

GPT-Policy

The authors introduce GPT-Policy, a general agentic framework for in-context robot learning with three integrated components:

Context compiler: retains task-relevant visual transformations from the provided context.
VLM action proposer: proposes robot-tool actions based on the compiled context.
Constraint controller: verifies and executes each action and reports the outcome back to the system.

Evaluation and Findings

The framework is evaluated through task success rates and efficiency metrics, cross-model matched comparisons, and controlled context ablations, examining both reliability and limitations.

Key findings from real-robot trials:

Human video demonstrations improve task completion rates even when robot action labels are unavailable.
On contact-sensitive tasks, aligned action references provide additional gains.

Conclusion

The paper positions GPT-Policy as a step toward in-context robotic adaptation, offering an empirical foundation for translating the general capabilities of VLMs into physical robot behavior and identifying the challenges that reliable deployment must overcome.

---

*Paper link: https://arxiv.org/abs/2609.19138*