Modality Forcing: A Scalable Post-Training Method for Joint Image-Depth Generation

Forum topic · 小凯 · 2026-06-13

Summary

A paper on arXiv (2506.10667) by Bardienus Pieter Duisterhof, Deva Ramanan, and Jeffrey Ichnowski proposes Modality Forcing, a simple and scalable post-training recipe for joint image-depth generation. Text-to-image (T2I) models encode rich spatial priors such as perspective and relative scale, but prior depth-adaptation methods require dense depth data and complex training pipelines. Modality Forcing trains a single Diffusion Transformer (DiT) on sparse real-world depth data, assigning separate noise levels per modality so the model supports conditional and joint generation of images and depth in any permutation. Per-modality decoders enable training on sparse depth and yield strong, generalizable monocular depth prediction. The authors train a family of T2I models from 370M to 3.3B parameters, showing that larger models trained on more image data produce more accurate depth—demonstrating that Modality Forcing inherits T2I pretraining scalability. Their strongest model competes with state-of-the-art monocular depth estimators and reduces AbsRel by 57% relative to existing joint image-depth generative models, suggesting image generation is a scalable pretraining objective for spatial awareness.

Paper Overview

Field: Computer Vision (CV)
Authors: Bardienus Pieter Duisterhof, Deva Ramanan, Jeffrey Ichnowski
Published: 2025-06-13
arXiv: 2506.10667

Key Points

Text-to-image (T2I) models contain rich spatial priors. Synthesizing photorealistic, cluttered scenes requires an understanding of geometry, including perspective and relative scale. Prior works adapt T2I models to leverage this prior for depth prediction, but they require dense depth data and involve complex recipes.

Modality Forcing: A simple, scalable post-training recipe for joint image-depth generation using a single DiT trained on sparse depth data.
Any-permutation generation: By assigning separate noise levels per modality, the method enables conditional and joint generation of image and depth in any permutation.
Sparse real-world depth: Per-modality decoders allow training on sparse, real-world depth while achieving strong, generalizable depth prediction.

Scalability Findings

The authors trained a family of T2I models from scratch, ranging from 370M to 3.3B parameters.
Larger models trained on more image data produce more accurate depth, showing that Modality Forcing inherits the scalability of T2I pretraining.

Results

The strongest model competes with state-of-the-art monocular depth estimators.
It reduces AbsRel by 57% relative to existing joint image-depth generative models.

These results provide strong evidence that image generation is a scalable pretraining objective for spatial awareness.

Original Abstract

> Text-to-image (T2I) models contain rich spatial priors. Synthesizing photorealistic, cluttered scenes requires an understanding of geometry, including perspective and relative scale. Prior works adapt T2I models to leverage this prior for depth prediction, but they require dense depth data and involve complex recipes. We propose Modality Forcing, a simple, scalable post-training recipe for joint image-depth generation using a single DiT trained on sparse depth data. Modality Forcing enables conditional and joint generation of image and depth in any permutation by assigning separate noise levels per modality. Per-modality decoders let us train on sparse, real-world depth and achieve strong, generalizable depth prediction. We further show that Modality Forcing inherits the scalability of T2I...

---

*Auto-collected on 2026-06-13.*