Paper Overview
Field: Computer Vision (CV) Authors: Pradhaan S Bhat, Naveen Chandra R, Rishubh Parihar Published: 2025-06-23 arXiv: 2506.18495
Summary
Text and 2D-conditioning interfaces provide weak, ambiguous control over spatial transformations in image editing — particularly under large object motions and camera changes. Prior work has used 3D primitives such as boxes, but only as loose conditioning signals indicating approximate object location rather than specifying the transformation.
This paper instead uses 3D boxes as structured specifications: the user provides the input and output boxes of the edit, casting editing as a well-posed geometry problem. This "thinking in boxes" interface, where each box face is color-coded to convey 3D orientation, gives precise control over translation, rotation, scaling, and viewpoint changes in real images while preserving scene and object identity, and recovering previously unseen object regions.
To ground transformations in scene appearance, the authors introduce a depth-aligned planar floor as a global reference frame, colored with depth-aware cues. Under this structural conditioning, the image generator produces consistent results under large transformations.
After two-stage training — on synthetic multi-object scenes and a small number of real-world videos from Objectron — the system generalizes to complex in-the-wild real images. The method operates directly on real photos and significantly outperforms recent state-of-the-art approaches on large-scale 3D editing.
Key Contributions
- Box-based edit specification: input/output 3D boxes define edits as a well-posed geometry problem instead of ambiguous text or loose conditioning
- Color-coded box faces to communicate 3D orientation for precise control of translation, rotation, scaling, and viewpoint changes
- Depth-aligned planar floor as a global reference frame grounded in scene appearance
- Identity preservation while synthesizing previously unseen object regions
- Two-stage training (synthetic scenes + Objectron real videos) enabling generalization to real in-the-wild images