[论文] Splat2Act: From Captured Scenes to Personalized VLA Manipulation

论文概要

研究领域: CV
作者: Xiaomeng Ye, Krishna Jaganathan, Gyubeum Lim, Gunjan Chhablani, Muhammad Zubair Irshad, Zsolt Kira
发布时间: 2026-10-09
arXiv: 2610.13945

中文摘要

用户希望把通用机器人操作策略带进家里或办公室，从架子、台面、桌子上拿起自己的物品。然而即便强如 π0.5，在采集的部署场景中也实现 0% 抓取成功率，且在每个用户环境采集真实示范的成本高得离谱。我们提出 Splat2Act：一条自动化 real-to-sim-to-real 流水线，仅凭手持手机扫描和用户物体的图像，即可将预训练的视觉-语言-动作（VLA）策略个性化到特定场景和物体集合。与以往只重建单个桌面不同，Splat2Act 处理楼宇级室内环境：用 3D 高斯泼溅构建照片级数字孪生与碰撞几何，自动发现桌面与非桌面容器，构造可行的抓取任务，并以特权仿真 oracle 生成成功示范。应用于 101 个校园扫描环境（中位可导航面积 104.87 平方米、7 个容器），产出 852 个容器和超过 68,000 条抓取示范。两阶段微调 π0.5 后，最强个性化策略在目标场景达 62.5% 总体抓取成功率；部署到真实 Stretch 3 机器人上，在 12 项任务子集上达 63.9%，与仿真中的 66.7% 接近，表明自动化 real-to-sim-to-real 训练仅凭仿真 oracle 数据即可实现 VLA 策略个性化。

原文摘要

A user who brings a generalist manipulation policy into their home or office wants it to pick up their objects from their shelves, counters, and tables. Yet even a strong generalist like \(π_{0.5}\) achieves 0% pickup success in our captured deployment scenes without adaptation, and collecting real demonstrations in every user's environment is prohibitively expensive. We present Splat2Act, an automated real-to-sim-to-real pipeline that personalizes a pretrained vision-language-action (VLA) policy to a specific scene and object set using only a handheld phone scan and images of the user's objects. Unlike prior real-to-sim work that reconstructs individual tabletops, Splat2Act operates on building-scale indoor environments: it builds a photorealistic digital twin with 3D Gaussian Splatting and...

自动采集于 2026-10-13

#论文 #arXiv #CV #小凯