[论文] Can Discrete Tokens Serve as Semantically Lossless Representations of ...

论文概要

研究领域: CV / 多模态
作者: Jialv Zou, Xiaotong Li, Zhengrong Yue, Xu Huang, Yaxin Luo, Wenhao Yan, Haodong Li, Wenyu Liu, Xinggang Wang
发布时间: 2026-10-27
arXiv: 2610.37662

中文摘要

离散视觉token以类似语言的方式表示视觉信息，为共享next-token预测目标下的多模态建模提供了原生接口。然而，量化误差被广泛认为会导致不可避免的语义损失，从根本上限制了离散视觉token的语义保真度。本文以多模态理解为语义保真度的代理指标重新审视这一假设，发现量化误差并不可靠——误差更大的离散表示反而可能实现更好的理解效果。真正决定下游性能的关键因素是离散索引的有效比特预算。但现有量化器最终会饱和：额外码本带来的语义信息越来越少。为此，本文提出分解向量量化（FVQ），用对称并行码本量化视觉特征的互补固定维度视图，在大比特预算下持续提升，最终在多模态理解上追平连续视觉表示。该模型在工业级多模态数据上训练，在九指标平均上超过UniAR和Emu3-Chat分别22.8%和59.2%，是首个在细粒度OCR等任务上可与Qwen3-VL等强连续表示模型竞争的离散表示模型。代码和模型已开源。

原文摘要

Discrete visual tokens represent vision in a discrete form analogous to language, offering a native interface for multimodal modeling under a shared next-token prediction objective. However, quantization error is widely assumed to cause unavoidable semantic loss, fundamentally limiting the semantic fidelity of discrete visual tokens and positioning continuous visual representations as an intrinsic upper bound. Using multimodal understanding as a proxy for semantic fidelity, we revisit this assumption and find quantization error unreliable, as discrete representations with substantially larger errors can achieve better understanding. Instead, the effective bit budget of discrete indices emerges as a key determinant of downstream performance. However, existing quantizers eventually saturate as additional codebooks contribute progressively less semantic information. As one approach to alleviating this saturation, we introduce Factorized Vector Quantization, which quantizes complementary, fixed-dimensional views of the visual feature with symmetric parallel codebooks. It continues to improve at larger bit budgets, ultimately matching continuous visual representations in multimodal understanding. Trained on industrial-scale multimodal data, our model achieves state-of-the-art performance among discrete-representation models, outperforming UniAR and Emu3-Chat by 22.8% and 59.2%, respectively, on a nine-metric average. To our knowledge, it is also the first discrete-representation model competitive with strong continuous-representation models such as Qwen3-VL, including on fine-grained OCR tasks. We further identify the key principles and a practical recipe enabling this competitiveness. These results and insights support the potential of discrete visual representations to underpin large-scale native multimodal foundation models. Code and models are available at https://github.com/hustvl/FVQ

自动采集于 2026-10-29

#论文 #arXiv #AI #小凯