Modality Forcing: Scalable Joint Image-Depth Generation from Text-to-Image Models

Forum topic · 小凯 · 2026-06-14

Summary

A paper on arXiv (2506.10667) by Bardienus Pieter Duisterhof, Deva Ramanan, and Jeffrey Ichnowski introduces Modality Forcing, a simple and scalable post-training method that enables a single diffusion transformer (DiT) to jointly generate images and depth maps. By assigning separate noise levels to each modality, the method supports conditional and joint generation in any permutation, and per-modality decoders allow training on sparse real-world depth data, yielding strong generalizable depth prediction. The authors show the approach inherits the scalability of text-to-image pretraining: training DiT models from 370M to 3.3B parameters, larger models trained on more image data produce more accurate depth. Their strongest model competes with state-of-the-art monocular depth estimators and reduces AbsRel by 57% compared to existing joint image-depth generation models, suggesting image generation is a scalable pretraining objective for spatial awareness.

Paper Overview

Field: Computer Vision Authors: Bardienus Pieter Duisterhof, Deva Ramanan, Jeffrey Ichnowski Published: 2025-06-13 arXiv: 2506.10667

English Translation (Abstract)

Text-to-image (T2I) models contain rich spatial priors. Synthesizing photorealistic, cluttered scenes requires an understanding of geometry, including perspective and relative scale. Prior works adapt T2I models to leverage this prior for depth prediction, but they require dense depth data and involve complex recipes. We propose Modality Forcing, a simple, scalable post-training recipe for joint image-depth generation using a single DiT trained on sparse depth data. Modality Forcing enables conditional and joint generation of image and depth in any permutation by assigning separate noise levels per modality. Per-modality decoders let us train on sparse, real-world depth and achieve strong, generalizable depth prediction. We further show that Modality Forcing inherits the scalability of T2I pretraining: training a family of T2I models from scratch (370M to 3.3B parameters), we find that larger models trained on more image data yield more accurate depth. Our strongest model competes with state-of-the-art monocular depth estimators, reducing AbsRel by 57% relative to existing joint image-depth generation models. These results provide strong evidence that image generation is a scalable pretraining objective for spatial awareness.

Key Contributions

Modality Forcing: assigns separate noise levels per modality, enabling conditional and joint generation of image and depth in any permutation with a single DiT.
Sparse-depth training: per-modality decoders allow training on sparse, real-world depth annotations rather than dense synthetic data.
Scalability: models scaled from 370M to 3.3B parameters show that depth accuracy improves with model size and data, inheriting T2I pretraining scaling behavior.
Results: the strongest model competes with state-of-the-art monocular depth estimators and reduces AbsRel by 57% versus existing joint image-depth generation models.

Conclusion

The results suggest that image generation itself can serve as a scalable pretraining objective for spatial (geometric) awareness.

---

*Auto-collected on 2026-06-14*