Paper Overview
Field: Computer Vision (CV) Authors: Hongyang Du, Yunfei Xie, Junjie Ye, Jiawei Yang, Xiaoyan Cong, Haodong Zhang, Yongchao Huang, Haiyu Wu, Zongxia Li, Shihang Gui, Dawei Liu, Runhao Li, Jingcheng Ni, Chen Wei, Randall Balestriero, Yue Wang Published: 2026-09-25 arXiv: 2609.31620
Summary
Representation autoencoders (RAEs) reuse features from a pretrained visual encoder as reconstruction and diffusion latents, integrating strong visual representations into image generation. However, RAEs still need to decide which encoder layers form the shared latent space for the generator and pixel decoder. This choice involves a trade-off:
- Shallower layers tend to preserve fine pixel details better.
- Deeper layers tend to yield better generation metrics.
- Flexible reconstruction: On ImageNet-256 with DINOv3-L, a single FuseReg decoder can reconstruct from full, sparse, and single-layer fusions without retraining, achieving higher PSNR than decoders trained specifically on fixed fusions.
- Improved generation: Simply swapping the decoder reduces unguided gFID by 27%, with the generator (RAEv2 DiT-XL) held unchanged.
- Extensible to diffusion training: Applying the same regularization principle as joint two-stage regularization reduces unguided gFID by 29% on DiT-Base.
A fixed heuristic layer fusion therefore couples two stages that benefit from different information.
The FuseReg Method
FuseReg replaces heuristic feature selection with training over random subsets of encoder layers. The authors theoretically analyze the underlying mechanism: subset sampling explicitly penalizes sensitivity to cross-layer disagreement, making downstream models robust to varying layer-fusion configurations.
Key Results
Takeaway
Training downstream models for layer-fusion robustness can narrow the reconstruction-generation gap without modifying the pretrained encoder.
Abstract (Original)
Representation autoencoders (RAEs) reuse features from a pretrained visual encoder as reconstruction and diffusion latents, integrating strong visual representations into image generation. However, RAEs still need to decide which encoder layers form the shared latent space for the generator and pixel decoder. This choice involves a trade-off. Shallower layers tend to preserve fine pixel details better, while deeper layers tend to yield better generation metrics. A fixed heuristic layer fusion therefore couples two stages that benefit from different information. We introduce FuseReg, which replaces heuristic feature selection with training over random subsets of encoder layers. We theoretically analyze the underlying mechanism: subset sampling explicitly penalizes sensitivity to cross-layer disagreement.
---
*Auto-collected on 2026-09-29.*