From RGB Generation to Dense Field Readout: ReChannel for Pixel-Space Dense Prediction with Pretrained DiTs

Paper Overview

• Field: Computer Vision
• Authors: Zanyi Wang, Xin Lin, Haodong Li
• Published: 2025-07-09
• arXiv: 2507.06828

Summary

Large-scale text-to-image models are attractive backbones for dense prediction because RGB generation pretraining learns rich semantic, structural, and geometric priors. Existing generative and editing approaches reuse these priors by casting dense prediction as target generation: annotations such as depth, normals, alpha mattes, masks, and heatmaps are encoded into an RGB-trained VAE latent space and decoded back as image-like targets.

The authors argue this inherits more of the generative output interface than dense prediction requires: unlike RGB synthesis, dense prediction asks for pixel-correct, task-native fields on the same image plane, not new RGB content to be rendered.

Key Observation

A pretrained DiT already organizes RGB inputs through a patch-to-token-to-patch lattice on the image plane. Each token indexes a fixed output patch whose channels can carry task-native quantities instead of RGB appearance.

Method: ReChannel

• Keep the VAE encoder that defines the DiT input distribution, but discard the target-side decoder.
• Attach task LoRA adapters to the frozen DiT.
• Map each token to its p×p×K_t pixel-space patch via a shared token-local linear head — roughly 33K parameters, with no spatial mixing.

Results

Evaluated with FLUX-Klein across six dense prediction tasks and over a dozen benchmarks:

• New SOTA on trimap-free matting, KITTI depth, and referring segmentation.
• Competitive on normals, saliency, and pose.
• In matched 4B settings, more accurate and 2.48× faster than editing-plus-latent-decoding counterparts.

Conclusion

Dense perception can benefit from generative pretraining without inheriting its generative output interface.

Original Abstract (excerpt)

Large-scale text-to-image models are attractive backbones for dense prediction because RGB generation pretraining learns rich semantic, structural, and geometric priors. Existing generative and editing approaches reuse these priors by casting dense prediction as target generation: annotations such as depth, normals, alpha mattes, masks, and heatmaps are encoded into an RGB-trained VAE latent space and decoded back as image-like targets. We argue this inherits more of the generative output interface than dense prediction requires: unlike RGB synthesis, dense prediction asks for pixel-correct, task-native fields on the same image plane, not new RGB content to be rendered. Our key observation is that a pretrained DiT already organizes RGB inputs through a patch-to-token-to-patch lattice on the image plane...

---

*Auto-collected on 2026-07-09*