Paper Overview
Research area: Computer Vision Authors: Bardienus Pieter Duisterhof, Deva Ramanan, Jeffrey Ichnowski, Justin Johnson, Keunhong Park Published: 2026-06-11 arXiv: 2606.13676
Summary
Text-to-image (T2I) models contain rich spatial priors. Synthesizing photorealistic, cluttered scenes requires an understanding of geometry, including perspective and relative scale. Prior works adapt T2I models to leverage this prior for depth prediction, but they require dense depth data and involve complex recipes. The authors propose Modality Forcing, a simple, scalable post-training recipe for joint image-depth generation using a single DiT trained on sparse depth data.
Key Points
- Modality Forcing enables conditional and joint generation of image and depth in any permutation by assigning separate noise levels per modality.
- Per-modality decoders allow training on sparse, real-world depth while achieving strong, generalizable depth prediction.
- The method inherits the scalability of T2I pretraining: a family of T2I models trained from scratch (370M to 3.3B parameters) shows that larger models trained on more image data produce more accurate depth.
- The strongest model is competitive with state-of-the-art monocular depth estimators and reduces AbsRel by 57% relative to existing joint image-depth generation models.
- These results provide strong evidence for image generation as a scalable pretraining objective for spatial awareness.
Original Abstract (excerpt)
> Text-to-image (T2I) models contain rich spatial priors. Synthesizing photorealistic, cluttered scenes requires an understanding of geometry, including perspective and relative scale. Prior works adapt T2I models to leverage this prior for depth prediction, but they require dense depth data and involve complex recipes. We propose Modality Forcing, a simple, scalable post-training recipe for joint image-depth generation using a single DiT trained on sparse depth data...
--- *Auto-collected on 2026-06-15*