Let ViT Speak: GenLIP — Generative Language-Image Pre-training for Vision Transformers

Forum topic · 小凯 · 2026-05-05

Summary

This paper introduces GenLIP (Generative Language-Image Pre-training), a minimalist pre-training framework that trains Vision Transformers (ViTs) to serve multimodal large language models (MLLMs). Instead of relying on contrastive batch construction or an extra text decoder, GenLIP trains a ViT to predict language tokens directly from visual tokens with a standard autoregressive language modeling objective. The authors highlight three advantages: simplicity (a single transformer jointly models visual and textual tokens), scalability with both data and model size, and competitive or superior performance across diverse multimodal benchmarks. Trained on 8B samples from Recap-DataComp-1B, GenLIP matches or exceeds strong baselines while using substantially less pre-training data. Continued pre-training on multi-resolution images at native aspect ratios further improves detail-sensitive tasks such as OCR and chart understanding, positioning GenLIP as a strong foundation for MLLM vision encoders.

Overview

Field: Computer Vision
Authors: Yan Fang, Mengcheng Lan, Zilong Huang, Weixian Lei, Yunqing Zhao, Yujie Zhong, Yingchen Yu, Qi She, Yao Zhao, Yunchao Wei
arXiv: 2605.00809

Key Points

GenLIP (Generative Language-Image Pre-training) is a minimalist generative pre-training framework for Vision Transformers (ViTs) targeting multimodal large language models (MLLMs).

To better align vision encoders with the autoregressive nature of LLMs, GenLIP trains a ViT to predict language tokens directly from visual tokens using a standard language modeling objective.
It requires no contrastive batch construction and no additional text decoder.

Three Key Advantages

1. Simplicity: a single transformer jointly models visual and textual tokens. 2. Scalability: it scales effectively with both data and model size. 3. Performance: it achieves competitive or superior results across diverse multimodal benchmarks.

Training and Results

Trained on 8B samples from Recap-DataComp-1B, GenLIP matches or surpasses strong baselines despite using substantially less pre-training data.
After continued pre-training on multi-resolution images at native aspect ratios, GenLIP further improves on detail-sensitive tasks such as OCR and chart understanding.

These results make GenLIP a strong foundation for vision encoders in MLLMs.

Original Abstract

In this paper, we present GenLIP, a minimalist generative pretraining framework for Vision Transformers (ViTs) designed for multimodal large language models (MLLMs). To better align vision encoders with the autoregressive nature of LLMs, GenLIP trains a ViT to predict language tokens directly from visual tokens using a standard language modeling objective, without contrastive batch construction or an additional text decoder. This design offers three key advantages: (1) Simplicity: a single transformer jointly models visual and textual tokens; (2) Scalability: it scales effectively with both data and model size; and (3) Performance: it achieves competitive or superior results across diverse multimodal benchmarks. Trained on 8B samples from Recap-DataComp-1B, GenLIP matches or surpasses strong baselines despite using substantially less pretraining data. After continued pretraining on multi-resolution images at native aspect ratios, GenLIP further improves on detail-sensitive tasks such as OCR and chart understanding, making it a strong foundation for vision encoders in MLLMs.