Thinking in Boxes: Making 3D Editing in Real Images Easy

> *Thinking in Boxes: 3D Editing in Real Images Made Easy* > > Authors: Pradhaan S Bhat, Naveen Chandra R, Rishubh Parihar | arXiv preprint, cs.CV

Key points

• Motivation. Traditional image editing offers poor control: cropping loses content, content-aware fill is unpredictable, and generative fill from text prompts is ambiguous. Text like "move grandma to the left" can be misinterpreted in many ways.
• Core idea. Replace fuzzy interfaces with 3D bounding boxes as an editing specification:
1. An input box surrounds the object to edit in the source image (drawn or auto-detected). 2. An output box defines the target position, rotation, scale, and viewpoint.
• Editing becomes a geometric problem: generate a visually coherent image consistent with the object's appearance under the output box.
• Color-coded orientation. Each box face maps to a distinct color corresponding to a 3D direction, so users specify orientation by "matching colors"—like a LEGO manual—rather than learning geometric terminology.

Why 3D editing is hard

• Occlusion. Moving an object requires inpainting revealed regions and generating new occlusions; in 3D these have deterministic explanations (geometry, camera, light) rather than being pure hallucination.
• Viewpoint change. Large pose/viewpoint edits are nearly impossible from a single 2D image without some 3D reasoning; boxes encode the initial and target poses explicitly.
• Limits of existing methods:
• *Dense processing* (large VLMs over whole images): strong but expensive and hard to control precisely.
• *Sparse conditioning* (2D scribbles, keypoints, text): cheap but weak control, prone to inconsistency.

Technical architecture

• Depth-aligned planar floor. A global reference frame defining "where the ground is," carrying depth-aware cues (shadows, perspective). All object transformations are specified relative to this floor, like a surveyor's baseline.
• Two-stage training (synthetic-to-real):
1. Synthetic multi-object scenes with perfect 3D ground truth for positions, poses, and shapes, across varied lighting and camera angles. 2. Real-world video (Objectron dataset) with 6DoF pose annotations, teaching realistic textures, materials, and lighting.
• Conditional image generator. Extracts the object's appearance from the input image, computes the target pose from the output box, and synthesizes the new view while handling occlusion, shadows, and lighting consistency.

Results

• Evaluated on 3D consistency, identity preservation, visual quality, and user studies.
• Compared to state-of-the-art methods, the approach shows significant improvements in 3D consistency and visual quality, particularly for large displacements and viewpoint changes, because it encodes 3D geometry directly instead of relying on 2D conditions.
• Failure cases: complex materials (transparent, reflective, hair/fur), heavily occluded scenes, and rare object categories in training data—pointing to future work in material modeling, scene understanding, and broader training data.

Broader significance

• Interface philosophy. The method bridges HCI's classic direct-manipulation vs. command-language trade-off: drawing and dragging boxes is intuitive, while the boxes' structure gives command-language precision.
• 2D-to-3D paradigm shift. Part of the field's progression from 2D image processing through 2.5D understanding and 3D reconstruction to 3D-aware generation (NeRF, 3D Gaussian Splatting, diffusion models). Instead of full 3D reconstruction, it proposes a lightweight 3D interface—users just "think in boxes."
• Relation to NeRF. NeRF fully reconstructs scenes for free novel-view rendering but needs multiple views and is expensive to edit; Thinking in Boxes needs only a single image plus user-specified boxes—precise, interactive editing. The two are complementary: NeRF for reconstruction and observation, boxes for editing and creation.

Conclusion

The author closes with a philosophical note: the ultimate goal of technology is to make machines understand human intuition rather than forcing humans to learn machine language. The bounding box, ubiquitous in human spatial cognition from toy blocks to warehouse logistics, becomes the bridge between how people think about space and how AI edits images.

References

1. Bhat, P. S., Chandra, R. N., & Parihar, R. (2026). *Thinking in Boxes: 3D Editing in Real Images Made Easy*. arXiv preprint. 2. Mildenhall, B., et al. (2020). NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis. *ECCV*, 405-421. 3. Avrahami, O., et al. (2023). Break-A-Scene: Extracting Multiple Concepts from a Single Image. *SIGGRAPH Asia*. 4. Brooks, T., et al. (2023). InstructPix2Pix: Learning to Follow Image Editing Instructions. *CVPR*, 18392-18402. 5. Barron, J. T., et al. (2023). Zip-NeRF: Anti-Aliased Grid-Based Neural Radiance Fields. *ICCV*. 6. Rombach, R., et al. (2022). High-Resolution Image Synthesis with Latent Diffusion Models. *CVPR*, 10684-10695. 7. Gao, C., et al. (2023). 3D Gaussian Splatting for Real-Time Radiance Field Rendering. *SIGGRAPH*, 1-12.