Scalable Visual Pretraining for Language Intelligence (arXiv:2607.09657)

Summary

This post introduces a research paper on arXiv (2607.09657) arguing that large foundation models need not be trained on text alone. While advances in large foundation models have been driven mainly by pretraining on massive text corpora, much knowledge is conveyed through visual representations: charts, typeset equations, and page layouts carry rich information that plain text cannot fully capture. Current pipelines typically convert visually rich sources into pure text, discarding these cues. The paper challenges the default assumption that language models must be trained purely on text, and presents a systematic study showing that pretraining on the same corpus in visual form consistently outperforms text-only pretraining. The authors conclude that visual pretraining is a scalable learner for foundation model intelligence and offers an efficient path toward scalable language intelligence. The paper is authored by Yiming Zhang, Zhonghan Zhao, Wenwei Zhang, and colleagues, with affiliations spanning CV/AI research groups, and was released on arXiv in July 2026. Full details are available via the linked arXiv page.