> Paper: Let ViT Speak: Generative Language-Image Pre-training > Authors: Yan Fang, Mengcheng Lan, Zilong Huang, Weixian Lei, Yunqing Zhao, Yujie Zhong, Yingchen Yu, Qi She, Yao Zhao, Yunchao Wei > arXiv: 2605.00809 | 2026-05-01
1. The AI That Understands but Cannot Speak
Imagine a foreign tourist in China who can read street signs and menus but cannot ask for directions. He "understands" but cannot "speak."
This is precisely the predicament of the traditional Vision Transformer (ViT).
ViT excels at vision tasks, but in multimodal large language models (MLLMs) it needs to be "aligned" with the language model — and that alignment process is often complex, inefficient, and unnatural.
2. The Tower of Babel Problem in Multimodal Pre-training
Contrastive methods (e.g., CLIP):
- Require large-batch construction
- Learn "similarity" rather than "generation"
- ViT learns to "match" but not to "describe"
- Typically need an extra text decoder
- More complex architecture, higher training cost
- Indirect information flow between vision and language modules
- Split images into visual tokens (standard ViT practice)
- But the training objective is neither image classification nor text matching
- Instead: given the visual tokens, predict the next language token
- Optimized with a standard autoregressive language modeling loss
- Contrastive learning: learn similarity first, then figure out generation
- Multi-stage training: pre-train vision first, then align language
Generative methods:
The result: ViT is like a tourist who only nods — it knows what you mean, but it cannot respond in its own words.
3. GenLIP: Let ViT Speak Directly
The paper proposes GenLIP (Generative Language-Image Pre-training), a minimalist generative pre-training framework:
Core idea: > Train ViT to directly predict language tokens from visual tokens, using the standard language modeling objective.
Design advantages: 1. Simplicity: a single Transformer jointly handles vision and language 2. Alignment: the visual encoder is inherently aligned with autoregressive language models 3. Scalability: no need for complex contrastive learning or extra decoders
It is like teaching that tourist not to "translate then speak," but to think and express directly in the new language.
4. Technical Details: How Visual Tokens "Speak" Language
Key innovations:
The result: ViT learns to describe what it sees — not through translation, but through direct generation.
5. The Feynman-Style Judgment: The Most Direct Path Is Often Best
> "If you think you understand something but can't explain it in simple language, you don't understand it."
Traditional approaches took a detour:
> "If the end goal is for ViT to 'speak,' why not just train it to speak?"
The most direct path is often the best one.
6. Takeaways for Builders
If you are building multimodal AI systems, ask yourself:
1. Does my visual encoder truly "understand" language? 2. Am I taking unnecessary detours in my training method? 3. Can I train the multimodal model with a more direct objective? 4. Is autoregressive generation the most natural cross-modal alignment?
GenLIP shows that sometimes the most elegant solution is not adding more components, but simplifying the objective — let the model directly do what you want it to do.
If you want it to speak from images, train it to speak from images. No translation, no intermediary, no complex architecture. Directness is power.