ViQ: Text-Aligned Visual Quantized Representations at Any Resolution

Forum topic · 小凯 · 2026-06-28

Summary

ViQ is a visual quantized representation framework proposed by researchers including Xumin Yu and Yongming Rao (arXiv:2606.27313) that balances semantics and low-level details in discrete visual tokens while supporting native-resolution inputs. The method splits quantization learning into two stages: text-aligned pretraining, which enriches a visual encoder with semantic supervision from a pretrained language model and enables native-resolution processing, and feature discretization, which introduces a proximal representation learning strategy to progressively compress the feature space and a position-aware head quantization mechanism for arbitrary resolutions. Experiments show ViQ is competitive with state-of-the-art multimodal encoders using continuous high-dimensional visual features while maintaining high-fidelity reconstruction, and multimodal training with ViQ tokens achieves 20%-70% speedups across different base LLMs and training schemes.

Paper Overview

Field: Computer Vision (CV)
Authors: Xumin Yu, Zuyan Liu, Zhenyu Yang, Yuhao Dong, Shengsheng Qian, Jiwen Lu, Han Hu, Yongming Rao
Published: 2026-06-25
arXiv: 2606.27313

Abstract (translated from Chinese summary)

Building a unified representation for text and vision is a natural goal, as it enables simpler multimodal modeling and more efficient training. However, representing images as discrete signals in the same way as text inevitably introduces severe information loss. Existing work struggles to balance low-level details and high-level semantics in discrete representations: reconstruction-oriented representations often lack semantic information, while semantically stronger features typically suffer from severe detail loss.

The authors propose ViQ, a Visual Quantized Representations framework designed to balance semantics and details in discrete representations while supporting native-resolution inputs, allowing it to serve as a unified, general discrete representation for arbitrary visual inputs. The approach divides quantization learning into two stages:

1. Text-aligned pretraining — leverages semantically rich supervision from a pretrained language model to enhance the visual encoder and enable native-resolution visual input processing. 2. Feature discretization — introduces a proximal representation learning strategy to progressively compress the feature space, plus a position-aware head quantization mechanism to flexibly handle arbitrary resolutions.

Extensive experiments on multimodal tasks show that ViQ is competitive with state-of-the-art multimodal vision encoders using continuous high-dimensional visual features, while maintaining high accuracy in low-level reconstruction. Multimodal training with visual quantized representations also yields substantial efficiency gains, achieving up to 20%–70% speedups across different base LLMs and training schemes.

Original Abstract (excerpt)

> A unified representation for text and vision is a natural pursuit, as it enables simpler multimodal modeling and more efficient training. However, representing images as discrete signals in the same way as text inevitably introduces severe information loss. Existing work struggles to balance low-level details and high-level semantics in discrete representations... We present ViQ, a Visual Quantized Representations framework, which is designed to balance semantics and details in discrete representations while supporting inputs at native resolutions.

---

*Auto-collected on 2026-06-28.*