Twins: Learning to Predict Unified Representations with Focal Loss for Multimodal Models

Forum topic · 小凯 · 2026-07-28

Summary

Twins is a unified continuous visual token space for multimodal models that supports both understanding and image generation. It is formed by channel-wise concatenating ViT features and VAE latents on the same token grid, keeping sequence length and attention cost unchanged. Jointly modeling this representation in a Diffusion Transformer reveals a severe optimization imbalance: the model fits semantic ViT components well but struggles to match the VAE latent distribution. The authors trace this imbalance to three heterogeneity sources—frequency bias, intrinsic dimensionality, and conditional versus unconditional uncertainty. They adapt focal regression objectives to flow matching, up-weighting high-error VAE dimensions to balance optimization. On ImageNet, this yields gFID improvements of up to 10.57 over naive MSE loss without classifier-free guidance, while remaining competitive on multimodal understanding benchmarks and improving reconstruction fidelity. Paper: arXiv 2607.22531.

Twins: Learn to Predict Unified Representations with Focal Loss

Research area: Computer Vision Authors: Kaixiong Gong, Xin Cai, Bin Lin, Hao Wang, Yunlong Lin, Mingzhe Zheng, Bohao Li, Jian-Wei Zhang, Miles Yang, Zhao Zhong, Liefeng Bo, Xiangyu Yue arXiv: 2607.22531

Overview

Unified multimodal models seek a shared visual token space that supports both multimodal understanding and image generation. Discrete methods unify the interface via a shared codebook, whereas continuous pipelines often rely on two disparate representations — semantic features (e.g., ViT) for understanding and low-level latents (e.g., VAE) for synthesis — resulting in mismatched latent spaces.

Key Ideas

Twins token space: A unified continuous token space formed by channel-wise concatenating ViT and VAE features on the same token grid, so sequence length is unchanged and attention cost does not increase.
Optimization imbalance: Jointly modeling Twins in a Diffusion Transformer exposes severe imbalance — the model fits the ViT component well but struggles to match the VAE latent distribution.
Three sources of heterogeneity: frequency bias, intrinsic dimensionality, and conditional alignment vs. conditional-independent uncertainty.
Focal loss for flow matching: The focal regression objective is adapted to flow matching, up-weighting VAE dimensions with larger errors to better balance optimization between ViT and VAE components.

Results

On ImageNet, the method achieves gFID improvements of up to 10.57 over naive MSE loss, without classifier-free guidance.
Twins remains competitive on multimodal understanding benchmarks and improves reconstruction fidelity, narrowing the gap between understanding-oriented and generation-oriented representations.

Original Abstract (excerpt)

> Unified multimodal models seek a shared visual token space that supports both multimodal understanding and image generation. Discrete methods unify the interface via a shared codebook, whereas continuous pipelines often rely on two disparate representations -- semantic features (e.g., ViT) for understanding and low-level latents (e.g., VAE) for synthesis -- resulting in mismatched latent spaces. We propose Twins, a unified continuous token space formed by channel-wise concatenating ViT and VAE features on the same token grid, so the sequence length is unchanged and attention cost does not increase. However, jointly modeling Twins in a Diffusion Transformer exposes a severe optimization imbalance...

--- *Auto-collected on 2026-07-28*