Splat2Act: Personalizing VLA Manipulation Policies from Captured Scene Scans

Forum topic · 小凯 · 2026-10-13

Summary

Splat2Act is an automated real-to-sim-to-real pipeline that personalizes pretrained vision-language-action (VLA) manipulation policies to a specific scene and object set using only a handheld phone scan and images of the user's objects. Unlike prior real-to-sim work limited to individual tabletops, it handles building-scale indoor environments by constructing photorealistic digital twins with 3D Gaussian Splatting, automatically discovering tabletop and non-tabletop containers, generating feasible pick tasks, and producing successful demonstrations with a privileged simulation oracle. Applied to 101 campus-scanned environments (median navigable area 104.87 square meters, 7 containers), the pipeline yielded 852 containers and over 68,000 grasp demonstrations. After two-stage fine-tuning of pi0.5, the best personalized policy achieved 62.5% overall pickup success in target scenes; deployed on a real Stretch 3 robot over a 12-task subset, it reached 63.9%, close to the 66.7% simulated performance. Results show automated real-to-sim-to-real training can personalize VLA policies using only simulation oracle data.

Paper Overview

Field: CV Authors: Xiaomeng Ye, Krishna Jaganathan, Gyubeum Lim, Gunjan Chhablani, Muhammad Zubair Irshad, Zsolt Kira Published: 2026-10-09 arXiv: 2610.13945

Chinese Abstract (translated)

Users want to bring generalist robot manipulation policies into their homes or offices to pick up their own objects from shelves, countertops, and tables. However, even a strong generalist like π0.5 achieves 0% pickup success in captured deployment scenes, and collecting real demonstrations in every user's environment is prohibitively expensive. We propose Splat2Act: an automated real-to-sim-to-real pipeline that personalizes a pretrained vision-language-action (VLA) policy to a specific scene and object set using only a handheld phone scan and images of the user's objects. Unlike prior work that reconstructs only individual tabletops, Splat2Act handles building-scale indoor environments: it uses 3D Gaussian Splatting to build a photorealistic digital twin with collision geometry, automatically discovers tabletop and non-tabletop containers, constructs feasible grasp tasks, and generates successful demonstrations with a privileged simulation oracle. Applied to 101 campus-scanned environments (median navigable area 104.87 square meters, 7 containers), it produced 852 containers and over 68,000 grasp demonstrations. After two-stage fine-tuning of π0.5, the strongest personalized policy achieved 62.5% overall pickup success in target scenes; deployed on a real Stretch 3 robot over a 12-task subset, it achieved 63.9%, close to the 66.7% in simulation, showing that automated real-to-sim-to-real training can personalize VLA policies using only simulation oracle data.

Original Abstract

A user who brings a generalist manipulation policy into their home or office wants it to pick up their objects from their shelves, counters, and tables. Yet even a strong generalist like $π_{0.5}$ achieves 0% pickup success in our captured deployment scenes without adaptation, and collecting real demonstrations in every user's environment is prohibitively expensive. We present Splat2Act, an automated real-to-sim-to-real pipeline that personalizes a pretrained vision-language-action (VLA) policy to a specific scene and object set using only a handheld phone scan and images of the user's objects. Unlike prior real-to-sim work that reconstructs individual tabletops, Splat2Act operates on building-scale indoor environments: it builds a photorealistic digital twin with 3D Gaussian Splatting and...

--- *Auto-collected on 2026-10-13*