[论文] From Pixels to Actions - Towards Native Vision-Language-Action Policie...

论文概要

研究领域: CV 作者: Weichen Fan, Haiwen Diao, Penghao Wu, Siyuan Hu, Ziqi Huang, Tianqi Liu, Mingze Sun, Ziyi Wang, Zhongang Cai, Hanming Deng, Quan Wang, Lei Yang, Lewei Lu, Fangzhou Hong, Zhaoxi Chen, Dahua Lin, Ziwei Liu 发布时间: 2026-11-04 arXiv: 2611.06918

中文摘要

视觉-语言-动作模型通常建立在一个简单假设之上：感知先由视觉编码器压缩，之后才开始动作推理。然而，机器人操作往往恰恰依赖于最容易在这种抽象中丢失的视觉信号——局部几何、边界、深度和细微的状态变化。这引出一个根本问题：动作策略应继承固定的视觉端点，还是直接访问不断演化的视觉计算？我们提出 NEO-VLA，一个结构性无编码器的视觉-语言-动作框架。图像由轻量级 patch 卷积层进行 tokenize，与语言信息在主干网络中联合演化，使基于 flow matching 的动作专家能利用不同深度上受指令条件化的视觉表征。这一简单改变揭示了一致模式：受控对比中，原生架构持续优于基于编码器的方案；全量微调可缩小但不能消除差距，单纯扩大规模也无法弥补。更引人注目的是，逐层探测显示，对动作至关重要的信号往往在最终视觉表征形成之前就已达到峰值。在各种操作场景和分布偏移下，NEO-VLA 实现了更强、更鲁棒的控制。这些结果表明，具身控制受益于跨越计算层级的视觉表征访问，而非固定的编码器输出。

原文摘要

Vision-language-action models are typically built on a simple premise: perception is first compressed by visual encoders, and action reasoning begins only afterwards. Yet manipulation often hinges on precisely the visual signals most vulnerable to this abstraction, e.g., local geometry, boundaries, depth, and subtle state changes. This raises a fundamental question: should action policies inherit a fixed visual endpoint, or directly access evolving visual computation? We introduce NEO-VLA, a structurally encoder-free vision-language-action framework. Images are tokenized by lightweight patch convolutional layers and evolve jointly with language throughout the backbone, allowing a flow-matching action expert to draw from instruction-conditioned visual representations at different depths. Th...

#论文 #arXiv #CV #小凯