Summary
NEO-VLA is a structurally encoder-free vision-language-action (VLA) framework that challenges the standard premise of compressing perception through a fixed visual encoder before action reasoning. Instead, images are tokenized by lightweight patch convolutional layers and evolve jointly with language throughout the backbone, letting a flow-matching action expert access instruction-conditioned visual representations at different depths. This design preserves visual signals critical to manipulation—local geometry, boundaries, depth, and subtle state changes—that are often lost in encoder abstractions. Controlled comparisons show native encoder-free architectures consistently outperform encoder-based counterparts; full fine-tuning narrows but does not eliminate the gap, and scale alone cannot compensate. Layer-wise probing reveals that action-critical signals often peak before final visual representations form. Across diverse manipulation scenarios and distribution shifts, NEO-VLA delivers stronger and more robust robot control, suggesting embodied control benefits from layered access to visual computation rather than fixed encoder outputs. (arXiv: 2611.06918)
Overview
- Research area: Computer Vision (CV)
- arXiv: 2611.06918
- Authors: Weichen Fan, Haiwen Diao, Penghao Wu, Siyuan Hu, Ziqi Huang, Tianqi Liu, Mingze Sun, Ziyi Wang, Zhongang Cai, Hanming Deng, Quan Wang, Lei Yang, Lewei Lu, Fangzhou Hong, Zhaoxi Chen, Dahua Lin, Ziwei Liu
Abstract
Vision-language-action models are typically built on a simple premise: perception is first compressed by visual encoders, and action reasoning begins only afterwards. Yet manipulation often hinges on precisely the visual signals most vulnerable to this abstraction, e.g., local geometry, boundaries, depth, and subtle state changes. This raises a fundamental question: should action policies inherit a fixed visual endpoint, or directly access evolving visual computation?
We introduce NEO-VLA, a structurally encoder-free vision-language-action framework. Images are tokenized by lightweight patch convolutional layers and evolve jointly with language throughout the backbone, allowing a flow-matching action expert to draw from instruction-conditioned visual representations at different depths.
Key Findings
- In controlled comparisons, the native encoder-free architecture consistently outperforms encoder-based schemes.
- Full fine-tuning can narrow but not eliminate the gap; simply scaling up does not compensate.
- Layer-wise probing shows that action-critical signals often peak before the final visual representation has formed.
- Across various manipulation scenarios and distribution shifts, NEO-VLA achieves stronger, more robust control.
These results suggest that embodied control benefits from access to visual representations across computational layers, rather than relying on a fixed encoder output.
This page is an English static mirror generated for search and AI citation.
It may be a full translation or structured summary of the Chinese original.
Canonical interactive discussion lives on the Chinese page:
https://zhichai.net/topic/178636786