Thinking in Boxes: Making 3D Editing of Real Images Easy

> *Thinking in Boxes: 3D Editing in Real Images Made Easy* > > Authors: Pradhaan S Bhat, Naveen Chandra R, Rishubh Parihar | arXiv (cs.CV)

The Editing Dilemma

Imagine a photographer with a treasured photo of a grandmother in her garden, wanting her moved slightly left so the fountain in the background is fully visible. Traditional options fall short:

1. Cropping — simple but loses content. 2. Content-aware fill — often looks smeared. 3. Manual repainting — hours of skilled work. 4. Generative fill — unpredictable; text prompts are too ambiguous to control precisely what moves, how, and where.

The core problem: it is hard to precisely control the *act* of editing. Text is vague; 2D scribbles are imprecise. The insight of "Thinking in Boxes" is to reframe editing as a 3D geometric problem: move the object from coordinate (0.3, 0.5) to (0.2, 0.5), keep it facing the camera, and let the background extend naturally.

The Box Metaphor

Humans naturally reason about objects using bounding boxes: you can mentally rotate, push, or flip a chair without seeing every screw. The paper formalizes this intuition as an editing interface:

1. Input box: draw (or auto-detect) a 3D box around the object to edit. 2. Output box: specify a target box defining the object's new position, rotation, size, and viewpoint.

Editing becomes a geometric problem: given the input image and output box, generate a visually coherent image where the object's appearance matches the input.

To make orientation specification intuitive, each box face gets a distinct color code corresponding to a 3D direction — users just "match colors" (like LEGO instructions) instead of learning geometric terminology.

Why 3D Editing Is Hard

• Occlusion: moving a coffee cup requires reconstructing the table it covered, generating newly occluded pixels, and handling shadows. In 2D these are "magic" hallucinations; in 3D they have deterministic explanations (what was hidden was hidden because the object sat between camera and surface).
• Viewpoint changes: in 2D this requires reconstructing the scene and re-rendering; the box framework makes it tractable by defining initial and target 3D poses.

Existing methods split into two camps:

• Dense processing (large VLMs over the whole image): good quality but computationally expensive and hard to control.
• Sparse conditioning (scribbles, keypoints, text): cheap but weakly controlled, prone to inconsistency.

Thinking in Boxes seeks a middle ground: precise geometric control via 3D boxes with computational efficiency.

Technical Architecture

• Depth-aligned planar floor: a global reference frame anchoring all 3D transformations, carrying depth-aware cues like shadows and perspective — analogous to a surveyor's baseline on a construction site.
• Two-stage training:
1. Synthetic multi-object scenes — perfect 3D ground truth for learning geometric relationships. 2. Real-world video (Objectron dataset) with 6DoF pose annotations — adapting to real textures, materials, and lighting.
• Conditional image generation: extract object appearance features from the input, compute the target pose from the output box, and generate a consistent new view handling occlusion and lighting.

Experiments and Results

Evaluated on 3D consistency, identity preservation, visual quality, and user studies:

• Significantly better 3D consistency and visual quality than state-of-the-art methods on large 3D edits (big displacements, rotations, viewpoint changes), where 2D-conditioned methods struggle.
• Honest failure analysis: complex materials (transparent, reflective, furry), heavily occluded regions, and rare object categories remain challenging — pointing to future work in material modeling, scene understanding, and training data.

Deeper Significance

• Interface philosophy: bridging direct manipulation (dragging boxes — intuitive but imprecise) and command languages (precise but ambiguous) into one interface.
• 2D-to-3D paradigm shift: from 2D image processing (1980s–90s), through 2.5D understanding and 3D reconstruction, to today's 3D-aware generation (NeRF, 3D Gaussian Splatting, diffusion models). Rather than full 3D reconstruction, this work offers a *lightweight 3D interface*.
• Comparison with NeRF: NeRF fully reconstructs a scene for multi-view rendering — high quality but multi-view-dependent, expensive, and hard to edit. Thinking in Boxes needs only a single image plus user-specified boxes — interactive and precisely controllable. They are complementary: NeRF for reconstruction and observation; boxes for editing and creation.

Conclusion

Editing becomes as natural as moving a real object: draw a box, drag and rotate it, and the system handles occlusion, background reconstruction, and lighting. The ultimate goal of the technology is to make machines understand human intuition rather than force humans to learn machine language. The box is not just a geometric primitive — it is the entry point of human spatial cognition, moving AI from "knowing the name" to "knowing the why."

References

1. Bhat, P. S., Chandra, R. N., & Parihar, R. (2026). *Thinking in Boxes: 3D Editing in Real Images Made Easy*. arXiv preprint. 2. Mildenhall, B., et al. (2020). NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis. *ECCV*, 405-421. 3. Avrahami, O., et al. (2023). Break-A-Scene: Extracting Multiple Concepts from a Single Image. *SIGGRAPH Asia*. 4. Brooks, T., et al. (2023). InstructPix2Pix: Learning to Follow Image Editing Instructions. *CVPR*, 18392-18402. 5. Barron, J. T., et al. (2023). Zip-NeRF: Anti-Aliased Grid-Based Neural Radiance Fields. *ICCV*. 6. Rombach, R., et al. (2022). High-Resolution Image Synthesis with Latent Diffusion Models. *CVPR*, 10684-10695. 7. Gao, C., et al. (2023). 3D Gaussian Splatting for Real-Time Radiance Field Rendering. *SIGGRAPH*, 1-12.