Paper Overview
- Field: Computer Vision (CV)
- Authors: Ellis Brown, Chris Hoang, Shengyi Qian, Shang-Wen Li, Mike Lewis, Marjan Ghazvininejad, Luke Zettlemoyer, Lucas Beyer, Mengye Ren, Jakob Verbeek, Saining Xie, Hu Xu
- Published: 2026-11-12
- arXiv: 2611.17334
- Visual priors are not necessary. An encoder trained jointly from scratch performs on par with a pretrained encoder, with no separate training stage required.
- Dedicated vision capacity is what matters. Encoder-free models still lag on visual understanding even when the backbone is scaled up to match the encoder's parameter count.
- Capacity need not sit before fusion. Adding vision-specific modules in the early layers of the backbone performs comparably to a pre-fusion encoder, while reducing time-to-first-token latency.
Summary
How multimodal language models should encode visual input remains an open question. Most models use a pretrained vision encoder, while some recent approaches train the encoder from scratch jointly with the language model, or remove the encoder entirely. Vision encoders bundle together three design choices:
1. A visual prior from upstream pretraining 2. Dedicated vision capacity 3. Pre-fusion placement of that capacity (i.e., before the multimodal backbone)
Because published models also differ in data, architecture, and compute, direct comparisons cannot isolate the effect of each choice. The authors disentangle them via a controlled framework: all models are pretrained from scratch on common data, then compared on visual question answering and language benchmarks at matched lifetime compute.
Key Findings
Abstract (Original)
How multimodal language models should encode visual input remains up for debate. Most models use a pretrained vision encoder, while some recent recipes train the encoder from scratch with the language model or remove it entirely. Vision encoders bundle together three design choices: a visual prior from upstream pretraining, dedicated vision capacity, and pre-fusion placement of that capacity (i.e., before the multimodal backbone). Because published models also differ in data, architecture, and compute, comparing them cannot isolate the effect of each choice. We disentangle these choices via a controlled framework where we pretrain all models from scratch on common data and compare them on visual question answering and language benchmarks at matched lifetime compute.
---
*Auto-collected on 2026-11-14.*