Paper Overview
- Field: CV / Multimodal
- Authors: Jialv Zou, Xiaotong Li, Zhengrong Yue, Xu Huang, Yaxin Luo, Wenhao Yan, Haodong Li, Wenyu Liu, Xinggang Wang
- Published: 2026-10-27
- arXiv: 2610.37662
- Trained on industrial-scale multimodal data, achieving state-of-the-art performance among discrete-representation models
- Outperforms UniAR and Emu3-Chat by 22.8% and 59.2% respectively on a nine-metric average
- First discrete-representation model competitive with strong continuous-representation models such as Qwen3-VL, including on fine-grained OCR tasks
- Identifies key principles and a practical recipe for enabling this competitiveness
Abstract (translated)
Discrete visual tokens represent vision in a discrete form analogous to language, offering a native interface for multimodal modeling under a shared next-token prediction objective. However, quantization error is widely assumed to cause unavoidable semantic loss, fundamentally limiting the semantic fidelity of discrete visual tokens and positioning continuous visual representations as an intrinsic upper bound.Using multimodal understanding as a proxy for semantic fidelity, the authors revisit this assumption and find quantization error unreliable: discrete representations with substantially larger errors can achieve better understanding. Instead, the effective bit budget of discrete indices emerges as a key determinant of downstream performance. However, existing quantizers eventually saturate as additional codebooks contribute progressively less semantic information.
To alleviate this saturation, the paper introduces Factorized Vector Quantization (FVQ), which quantizes complementary, fixed-dimensional views of the visual feature with symmetric parallel codebooks. It continues to improve at larger bit budgets, ultimately matching continuous visual representations in multimodal understanding.
Key Results
*Auto-collected on 2026-10-29.*