VGGT-Edit: Feed-forward Native 3D Scene Editing with Residual Fields

Forum topic · 小凯 · 2026-05-17

Summary

VGGT-Edit is a feed-forward framework for text-conditioned, native 3D scene editing introduced by Kaixin Zhu and colleagues (arXiv 2605.15186). While generalizable feed-forward 3D reconstruction models excel at static scene perception, existing editing approaches rely on 2D-lifting strategies that edit views independently and lift them back to 3D, often producing blurry textures and inconsistent geometry. VGGT-Edit addresses this with depth-synchronized text injection, which aligns semantic guidance with the backbone's spatial pose encoding for stable instruction grounding. A residual transformation head then directly predicts 3D geometric displacements to deform the scene while keeping the background stable. Training uses a multi-term objective enforcing geometric accuracy and cross-view consistency. The authors also present DeltaScene, a large-scale dataset generated via an automated pipeline with 3D-consistency filtering. Experiments show VGGT-Edit substantially outperforms 2D-lifting baselines, yielding sharper object details, stronger multi-view consistency, and near-instant inference speeds, making it well suited for interactive 3D applications.

Overview

Research area: Computer Vision Authors: Kaixin Zhu, Yiwen Tang, Yifan Yang, Renrui Zhang, Bohan Zeng, Ziyu Guo, Ruichuan An, Zhou Liu, Qizhi Chen, Delin Qu, Jaehong Yoon, Wentao Zhang Published: 2026-05-14 arXiv: 2605.15186

Abstract

High-quality 3D scene reconstruction has recently advanced to generalizable feed-forward architectures capable of generating complex environments in a single forward pass. However, despite their excellence in static scene perception, these models remain limited in responding to dynamic human instructions, which restricts their use in interactive applications.

Existing editing methods typically rely on 2D-lifting strategies, where individual views are edited independently and then lifted back into 3D space. This indirect pipeline often results in blurry textures and inconsistent geometry, because 2D editors lack the spatial awareness needed to preserve structure across viewpoints.

To address these limitations, the authors propose VGGT-Edit, a feed-forward framework for text-conditioned native 3D scene editing. Key contributions:

Depth-synchronized text injection: aligns semantic guidance with the backbone's spatial pose encoding, ensuring stable instruction grounding.
Residual transformation head: directly predicts 3D geometric displacements to deform the scene while keeping the background stable.
Multi-term objective function: enforces geometric accuracy and cross-view consistency for high-fidelity results.
DeltaScene dataset: a large-scale dataset generated through an automated pipeline with 3D-consistency filtering to ensure realistic quality.

Results

Experiments show that VGGT-Edit substantially outperforms 2D-lifting baselines, producing sharper object details, stronger multi-view consistency, and near-instant inference speeds.

*Auto-collected on 2026-05-17.*