[论文] Z-sight: Unifying Vision-Language-Action and Latent World Modeling
研究领域: CV 作者: Junjie Ye, Yiqi Zhao, Muchen Xu 发布时间: 2025-11-15 arXiv: 2511.13002
论文概要
研究领域: CV 作者: Junjie Ye, Yiqi Zhao, Muchen Xu 发布时间: 2025-11-15 arXiv: 2511.13002
中文摘要
我们提出Z-sight——一个统一的机器人基础模型,将视觉-语言-动作(VLA)与生成式潜在世界建模耦合在一起。基于视频的世界动作模型(WAM)展示了联合未来预测与动作生成对控制的价值,但继承了视频生成器的高昂成本。最近的潜在空间策略避免了像素生成,但通常对齐中间特征或依赖冻结的目标编码器,使未来预测与VLM的原生特征空间解耦。Z-sight则直接在VLM感知当前时刻的视觉token空间中预测未来。通过将前瞻专家和连续动作专家与VLM以混合Transformer结构耦合,Z-sight在推理时联合去噪未来空间token和动作,无需像素重建或外部编码器。在人类视频和机器人轨迹上训练后,Z-sight在SIMPLE基准上达到81.9%的成功率(比最强基线高9.6个百分点),在六个Unitree G1任务上达到73.3%(高20.0个百分点),每块推理时间仅110毫秒,同时在桌面操作基准上保持竞争力。
原文摘要
We introduce Z-sight, a unified robot foundation model that couples vision-language-action (VLA) and generative latent world modeling. Video-based world action models (WAMs) show the value of joint future-and-action generation for control, yet inherit the heavy cost of video generators. Recent latent-space policies avoid pixel generation, yet typically align intermediate features or rely on frozen target encoders, decoupling future prediction from the VLM's native feature space. Z-sight instead predicts the future directly in the visual token space in which its VLM perceives the present. Coupling a foresight expert and a continuous action expert with the VLM in a mixture-of-transformers, Z-sight jointly denoises future spatial tokens and actions at inference without pixel reconstruction or...
*自动采集于 2026-11-20*
#论文 #arXiv #CV #小凯