φ-RIE: Converting Photorealistic 3D Gaussian Splatting Reconstructions into Interactive Simulation Environments

Forum topic · 小凯 · 2026-09-24

Summary

φ-RIE (arXiv:2609.26795) is a Gaussian-native pipeline that turns photorealistic 3D Gaussian Splatting (3DGS) scene reconstructions into physically interactive robot simulation environments. While 3DGS reconstructs captured scenes with high visual fidelity, the representation does not support object-level physical interaction—objects cannot move independently, make contact, or reveal occluded surroundings. φ-RIE addresses this by coupling asset construction with source scene removal: a single object identity defines both the movable simulator asset and the scene content to be removed and completed. Scene observation provides shared evidence for producing registered assets and completed background Gaussians, preserving unedited Gaussians while aligning visual and physical states. On 50 ScanNet++ scenes, evidence-based selection and registration retries improved matched F1 at 20mm from 0.336 to 0.383 at fixed retention. Additional evaluations demonstrate asset executability, manipulation gains over single-generator baselines, and the visual cost of conversion.

Paper Overview

Field: Computer Vision (CV)
Authors: Runyi Yang, Deheng Zhang, Xiaoye Wang, Kanzhi Wu, Lei Sun et al.
Date: 2026-09-22
arXiv: 2609.26795

Abstract

3D Gaussian Splatting (3DGS) can reconstruct a captured scene photorealistically, but the resulting representation does not by itself support physical interaction. Robot simulation instead requires object-level change—objects must move independently, make contact, and reveal previously occluded surroundings. This gap arises because object appearance may remain entangled with the background, while hidden object geometry and occluded background content may be unobserved.

To address this challenge, the authors present φ-RIE, a Gaussian-native pipeline that converts selected objects into movable simulator assets while preserving the remaining reconstruction.

Key Idea

The key observation is that asset construction and source removal should be coupled: one object identity should define both the movable asset and the scene content to remove and complete.

Scene Observation supplies shared evidence to Coupled Scene Construction
This produces registered assets and completed background Gaussians for simulator-driven rendering in an Interactive Environment
The coupling preserves unedited Gaussians while aligning visual and physical state

Results

On 50 ScanNet++ scenes, evidence-based selection and registration retry increase matched F1 at 20mm from 0.336 to 0.383 at fixed retention
Further tests demonstrate asset executability, manipulation gains over a single-generator baseline, and the visual cost of conversion

Together, these results demonstrate that φ-RIE enables interactive scene conversion.

---

*Auto-collected on 2026-09-24*