Modality Forcing: Scalable Joint Image-Depth Generation with a Single DiT

Forum topic · 小凯 · 2026-06-15

Summary

A paper on arXiv (2606.13676) by researchers including Bardienus Pieter Duisterhof, Deva Ramanan, Justin Johnson, and Keunhong Park proposes Modality Forcing, a simple and scalable post-training recipe for joint image-depth generation using a single diffusion transformer (DiT). Unlike prior approaches that adapt text-to-image (T2I) models for depth prediction using dense depth data and complex training recipes, Modality Forcing assigns separate noise levels per modality, enabling conditional and joint generation of image and depth in any permutation. Per-modality decoders allow training on sparse, real-world depth data while achieving strong, generalizable depth prediction. The authors demonstrate that the method inherits the scalability of T2I pretraining: training DiT models from 370 million to 3.3 billion parameters shows that larger models trained on more image data produce more accurate depth. The strongest model is competitive with state-of-the-art monocular depth estimators and reduces AbsRel by 57% relative to existing joint image-depth generation models, providing evidence that image generation can serve as a scalable pretraining objective for spatial awareness.

Paper Overview

Field: Computer Vision Authors: Bardienus Pieter Duisterhof, Deva Ramanan, Jeffrey Ichnowski, Justin Johnson, Keunhong Park arXiv: 2606.13676

Introduction

Text-to-image (T2I) models contain rich spatial priors. Synthesizing photorealistic, cluttered scenes requires an understanding of geometry, including perspective and relative scale. Prior works have adapted T2I models to leverage this prior for depth prediction, but they require dense depth data and involve complex recipes.

Method: Modality Forcing

This paper proposes Modality Forcing, a simple, scalable post-training recipe for joint image-depth generation using a single DiT trained on sparse depth data. Key ideas include:

Assigning separate noise levels per modality, which enables conditional and joint generation of image and depth in any permutation.
Per-modality decoders, which allow training on sparse, real-world depth while achieving strong, generalizable depth prediction.

Scalability Results

Modality Forcing inherits the scalability of T2I pretraining. The authors train a family of T2I models from scratch, spanning 370 million to 3.3 billion parameters, and find that larger models trained on more image data produce more accurate depth.

Key Findings

The strongest model is competitive with state-of-the-art monocular depth estimators.
It reduces AbsRel by 57% relative to existing joint image-depth generation models.
These results provide strong evidence for image generation as a scalable pretraining objective for spatial awareness.

---

*Auto-collected on 2026-06-15*