Twins: Learning to Predict Unified Representations with Focal Loss

Forum topic · 小凯 · 2026-07-28

Summary

Twins is a unified continuous visual token space for multimodal models that supports both understanding and image generation. It is formed by channel-wise concatenating ViT semantic features with VAE latents on the same token grid, keeping sequence length and attention cost unchanged. Jointly modeling these heterogeneous features in a Diffusion Transformer reveals a severe optimization imbalance: the model fits the ViT component well but struggles to match the VAE latent distribution. The authors trace this imbalance to three sources of heterogeneity: frequency bias, intrinsic dimension, and conditional alignment versus conditional independence uncertainty. They adapt a focal regression objective to flow matching, upweighting high-error VAE dimensions to rebalance optimization. On ImageNet this yields a gFID improvement of up to 10.57 over naive MSE loss without classifier-free guidance, while remaining competitive on multimodal understanding benchmarks and improving reconstruction fidelity.

Paper Overview

Field: Computer Vision (CV) Authors: Kaixiong Gong, Xin Cai, Bin Lin, Hao Wang, Yunlong Lin, Mingzhe Zheng, Bohao Li, Jian-Wei Zhang, Miles Yang, Zhao Zhong, Liefeng Bo, Xiangyu Yue Published: 2026-07-24 arXiv: 2607.22531

Original Abstract

Unified multimodal models seek a shared visual token space that supports both multimodal understanding and image generation. Discrete methods unify the interface via a shared codebook, whereas continuous pipelines often rely on two disparate representations -- semantic features (e.g., ViT) for understanding and low-level latents (e.g., VAE) for synthesis -- resulting in mismatched latent spaces. We propose Twins, a unified continuous token space formed by channel-wise concatenating ViT and VAE features on the same token grid, so the sequence length is unchanged and attention cost does not increase. However, jointly modeling Twins in a Diffusion Transformer exposes a severe optimization imbalance: the model fits the ViT component well but struggles to match the VAE latent distribution. We trace this imbalance to three sources of heterogeneity: frequency bias, intrinsic dimension, and conditional alignment versus conditional independence uncertainty. To address it, we adapt a focal regression objective to flow matching, weighting the high-error VAE dimensions and better balancing optimization between the ViT and VAE components. On ImageNet, this achieves a gFID improvement of up to 10.57 over naive MSE loss without classifier-free guidance. Twins is also competitive on multimodal understanding benchmarks and improves reconstruction fidelity, narrowing the gap between understanding-oriented and generation-oriented representations.

Key Contributions

Unified continuous token space: Channel-wise concatenation of ViT and VAE features on the same token grid, with no increase in sequence length or attention cost.
Analysis of optimization imbalance: Identifies frequency bias, intrinsic dimension, and conditional alignment/independence uncertainty as sources of heterogeneity when jointly modeling semantic and low-level features.
Focal loss for flow matching: Adapts focal regression to upweight hard-to-fit VAE dimensions.
Results: Up to 10.57 gFID improvement on ImageNet over naive MSE loss (without CFG), competitive multimodal understanding performance, and better reconstruction fidelity.