VGGT-Edit: Feed-Forward Native 3D Scene Editing with Residual Fields

Paper Overview

• Field: Computer Vision
• Authors: Kaixin Zhu, Yiwen Tang, Yifan Yang, Renrui Zhang, Bohan Zeng, Ziyu Guo, Ruichuan An, Zhou Liu, Qizhi Chen, Delin Qu, Jaehong Yoon, Wentao Zhang
• arXiv: 2605.15186

Summary

High-quality 3D scene reconstruction has recently advanced to generalizable feed-forward architectures capable of generating complex environments in a single forward pass. However, despite their strong static scene perception, these models remain limited in responding to dynamic human instructions, restricting their use in interactive applications.

Existing editing methods typically rely on 2D-lifting strategies, where individual views are edited independently and then lifted back into 3D space. This indirect pipeline often leads to blurry textures and inconsistent geometry, because 2D editors lack the spatial awareness required to preserve structure across viewpoints.

To address these limitations, the authors propose VGGT-Edit, a feed-forward framework for text-conditioned native 3D scene editing. Key contributions:

• Depth-synchronized text injection: aligns semantic guidance with the backbone's spatial pose alignment, ensuring stable instruction grounding.
• Residual transformation head: processes the semantic signal and directly predicts 3D geometric displacements to deform the scene while keeping the background stable.
• Multi-term objective: supervises training to enforce geometric precision and cross-view consistency for high-fidelity results.
• DeltaScene dataset: a large-scale dataset generated through an automatic pipeline with 3D-consistency filtering to ensure realistic quality.

Experiments show that VGGT-Edit substantially outperforms 2D-lifting baselines, producing sharper object details, stronger multi-view consistency, and near-instant inference speed.

*Auto-collected on 2026-05-17.*