ReChannel: From RGB Generation to Dense Field Readout with Frozen DiTs

Forum topic · 小凯 · 2026-07-09

Summary

Researchers propose ReChannel, a minimal output interface that adapts large text-to-image diffusion transformers (DiTs) for dense prediction tasks such as depth, normals, matting, and segmentation. Instead of encoding dense targets into an RGB-trained VAE latent space and decoding them as image-like outputs, ReChannel exploits the DiT's patch-to-token-to-patch lattice: each token indexes a fixed output patch whose channels can carry task-native quantities rather than RGB appearance. The method retains the VAE encoder, discards the target-side decoder, attaches task-specific LoRA adapters to the frozen DiT, and maps each token to its p×p×K_t pixel-space patch via a shared token-local linear head (~33K parameters, no spatial mixing). Using FLUX-Klein, ReChannel sets new state-of-the-art results on trimap-free alpha matting, KITTI depth, and referring segmentation, remains competitive on normals, saliency, and pose, and at a matched 4B setting is more accurate and 2.48× faster than edit-then-latent-decode baselines. Paper: arXiv 2507.06828.

Paper Overview

Field: Computer Vision Authors: Zanyi Wang, Xin Lin, Haodong Li Published: 2025-07-09 arXiv: 2507.06828

Abstract (Full Translation)

Large-scale text-to-image models are attractive backbones for dense prediction because RGB generation pretraining learns rich semantic, structural, and geometric priors. Existing generative and editing approaches reuse these priors by casting dense prediction as target generation: annotations such as depth, normals, alpha mattes, masks, and heatmaps are encoded into an RGB-trained VAE latent space and decoded back as image-like targets. The authors argue this inherits more of the generative output interface than dense prediction requires: unlike RGB synthesis, dense prediction asks for pixel-correct, task-native fields on the same image plane, not new RGB content to be rendered.

The key observation is that a pretrained DiT already organizes RGB inputs through a patch-to-token-to-patch lattice on the image plane, so every token indexes a fixed output patch whose channels can carry task-native quantities rather than RGB appearance. This is instantiated as ReChannel: keep the VAE encoder of the DiT's input distribution, but discard the target-side decoder; attach a task LoRA to the frozen DiT; and map every token to its p×p×K_t pixel-space patch through a shared token-local linear head — roughly 33K parameters, with no spatial mixing.

Using FLUX-Klein, the method is evaluated on six dense prediction tasks across a dozen benchmarks. This minimal interface sets new state-of-the-art on trimap-free alpha matting, KITTI depth, and referring segmentation, while remaining competitive on normals, saliency, and pose. At a matched 4B setting, it is more accurate than its edit-then-latent-decode counterpart and 2.48× faster — showing that dense perception can benefit from generative pretraining without inheriting its output interface.

Key Takeaways

Dense prediction does not need the full generative output interface (VAE decode of image-like targets).
A token-local linear readout head (~33K params) suffices to produce task-native dense fields.
New SOTA on matting, KITTI depth, and referring segmentation; 2.48× speedup over editing-based baselines.

---

*Auto-collected on 2026-07-09.*