From RGB Generation to Dense Field Readout: Pixel-Space Dense Prediction with ReChannel (arXiv 2507.06828)

Forum topic · 小凯 · 2026-07-09

Summary

A paper by Zanyi Wang, Xin Lin, and Haodong Li (arXiv 2507.06828, July 2025) argues that dense prediction should not inherit the full generative output interface of text-to-image models. While existing approaches encode depth, normals, mattes, masks, and heatmaps into an RGB-trained VAE latent space and decode them back as image-like targets, the authors note that a pretrained DiT already organizes inputs through a patch-to-token-to-patch lattice, so each token can map directly to a fixed output patch carrying task-native quantities instead of RGB appearance. They instantiate this as ReChannel: the VAE encoder is retained to preserve the DiT input distribution, the target-side decoder is discarded, a task LoRA adapts the frozen DiT, and a shared token-local linear head (~33K parameters, no spatial mixing) maps each token to its p×p×K_t pixel-space patch. Using FLUX-Klein, ReChannel sets new state of the art on trimap-free matting, KITTI depth, and referring segmentation, stays competitive on normals, saliency, and pose, and is 2.48× faster and more accurate than editing-plus-latent-decode baselines at matched 4B settings.

Paper Overview

Field: Computer Vision Authors: Zanyi Wang, Xin Lin, Haodong Li Published: 2025-07-09 arXiv: 2507.06828

Key Idea

Large-scale text-to-image models are attractive backbones for dense prediction because RGB generation pretraining learns rich semantic, structural, and geometric priors. Existing generative and editing approaches reuse these priors by casting dense prediction as target generation: annotations such as depth, normals, alpha mattes, masks, and heatmaps are encoded into an RGB-trained VAE latent space and decoded back as image-like targets.

The authors argue this inherits more of the generative output interface than dense prediction requires. Unlike RGB synthesis, dense prediction asks for pixel-correct, task-native fields on the same image plane — not new RGB content to be rendered.

Method: ReChannel

The key observation is that a pretrained DiT already organizes RGB inputs through a patch-to-token-to-patch lattice: each token indexes a fixed output patch whose channels can carry task-native quantities rather than RGB appearance. ReChannel instantiates this:

Retain the VAE encoder to preserve the DiT input distribution; discard the target-side decoder
Adapt the frozen DiT with a task LoRA
Map each token to its p×p×K_t pixel-space patch via a shared token-local linear head — about 33K parameters, with no spatial mixing

Results

Evaluated with FLUX-Klein across six dense prediction tasks and a dozen benchmarks:

New state of the art: trimap-free matting, KITTI depth, and referring segmentation
Competitive: surface normals, saliency, and pose
At matched 4B settings, more accurate and 2.48× faster than editing-plus-latent-decode counterparts

Conclusion

Dense perception can benefit from generative pretraining without inheriting its output interface.

--- *Auto-collected 2026-07-09*