Paper Overview
Field: Computer Vision Authors: Xumin Yu, Zuyan Liu, Zhenyu Yang, Yuhao Dong, Shengsheng Qian, Jiwen Lu, Han Hu, Yongming Rao Published: 2026-06-25 arXiv: 2606.27313
Abstract
A unified representation for text and vision is a natural pursuit, as it enables simpler multimodal modeling and more efficient training. However, representing images as discrete signals in the same way as text inevitably introduces severe information loss. Existing work struggles to balance low-level details and high-level semantics in discrete representations: reconstruction-oriented representations often lack semantic information, whereas semantically stronger features typically suffer from severe loss of detail.
We present ViQ, a Visual Quantized Representations framework, which is designed to balance semantics and details in discrete representations while supporting inputs at native resolutions, thereby enabling it to serve as a unified and general discrete representation for arbitrary visual inputs.
Method
Our approach divides quantization learning into two stages: text-aligned pretraining and feature discretization.
1. Text-aligned pretraining: We enhance the visual encoder with the semantically rich supervision of pretrained language models, and enable it to process native-resolution visual inputs. 2. Feature discretization: We propose a proximal representation learning strategy to progressively compress the feature space, along with a position-aware head quantization mechanism to flexibly handle arbitrary resolutions.
Results
Extensive experiments on multimodal tasks demonstrate that ViQ is competitive with state-of-the-art multimodal visual encoders using continuous high-dimensional visual features, while maintaining high accuracy in low-level reconstruction. We also show that multimodal training with visual quantized representations greatly improves efficiency, achieving speedups of up to 20%-70% across different base LLMs and training schemes.
--- *Auto-collected on 2026-06-28*