ReChannel: From RGB Generation to Dense Field Readout for Pixel-Space Dense Prediction

Forum topic · 小凯 · 2026-07-09

Summary

This paper (arXiv:2507.06828) by Zanyi Wang, Xin Lin, and Haodong Li argues that reusing large text-to-image models for dense prediction should not inherit the generative RGB output interface. Instead of encoding targets like depth, normals, or masks into an RGB-trained VAE latent space and decoding them as images, the authors exploit the fact that a pretrained DiT already maps patches to tokens and back on the image lattice. Their method, ReChannel, keeps the VAE encoder, discards the target-side decoder, attaches task LoRA adapters to the frozen DiT, and uses a token-local linear head (~33K parameters, no spatial mixing) to map each token to a p×p×K_t pixel-space patch. Evaluated with FLUX-Klein across six dense prediction tasks and a dozen benchmarks, it sets new SOTA on trimap-free matting, KITTI depth, and referring segmentation, stays competitive on normals, saliency, and pose, and is 2.48x faster and more accurate than edit-then-latent-decode baselines at matched 4B scale.

Paper Overview

Field: Computer Vision
Authors: Zanyi Wang, Xin Lin, Haodong Li
Published: 2025-07-09
arXiv: 2507.06828

Summary

Large-scale text-to-image models are attractive backbones for dense prediction because RGB generation pretraining learns rich semantic, structural, and geometric priors. Existing generative and editing approaches reuse these priors by casting dense prediction as target generation: annotations such as depth, normals, alpha mattes, masks, and heatmaps are encoded into an RGB-trained VAE latent space and decoded back as image-like targets.

The authors argue this inherits more of the generative output interface than dense prediction requires: unlike RGB synthesis, dense prediction asks for pixel-correct, task-native fields on the same image plane, not new RGB content to be rendered.

Their key observation is that a pretrained DiT already organizes RGB inputs through a patch-to-token-to-patch lattice on the image plane, so each token indexes a fixed output patch whose channels can carry task-native quantities rather than RGB appearance.

Method: ReChannel

Keep the VAE encoder that preserves the DiT input distribution, but discard the target-side decoder.
Attach task LoRA adapters to the frozen DiT.
Map each token to its p×p×K_t pixel-space patch through a shared, token-local linear head — roughly 33K parameters, with no spatial mixing.

Results

Evaluated with FLUX-Klein across six dense prediction tasks and over a dozen benchmarks:

New state of the art on trimap-free matting, KITTI depth, and referring segmentation.
Competitive performance on surface normals, saliency, and pose.
At a matched 4B setting, more accurate and 2.48x faster than editing-plus-latent-decoding counterparts.

Conclusion

Dense perception can benefit from generative pretraining without inheriting its generative output interface.

> Large-scale text-to-image models are attractive backbones for dense prediction because RGB generation pretraining learns rich semantic, structural, and geometric priors...

Read the full paper: arXiv:2507.06828

ReChannel: From RGB Generation to Dense Field Readout for Pixel-Space Dense Prediction

Summary

Paper Overview

Summary

Method: ReChannel

Results

Conclusion

Tags