Tencent and CUHK Find the Optimal Recipe for Native Bilingual Multimodal Models: Scaling From Scratch

Forum topic · ✨步子哥 · 2026-08-03

Summary

A paper by researchers at the Chinese University of Hong Kong and Tencent, 'Scaling Native Multimodal Pre-Training From Scratch' (arXiv:2607.22043), presents the first systematic scaling laws for training natively multimodal large language models without a vision encoder. Using MoE-based IsoFLOP experiments on 250B text tokens plus 75B multimodal tokens, the authors show that language and multimodal objectives follow distinct scaling laws: language optimal allocation is nearly insensitive to data composition, while multimodal allocation depends strongly on the multimodal data ratio. The paper maps a Pareto frontier over total compute, model size, token count, and data mix, providing an engineering blueprint for native multimodal pre-training. Notably, native multimodal training improves pure-text spatial reasoning, suggesting positive transfer rather than capacity competition. Limitations include training-loss-only validation, no open-source code, and findings specific to MoE architectures.

This article analyzes Scaling Native Multimodal Pre-Training From Scratch

Paper: Scaling Native Multimodal Pre-Training From Scratch
Authors: Haoyuan Wu, Aoqi Wu, Hai Wang, Jiajia Wu, Jinxiang Ou, Bei Yu
Institutions: The Chinese University of Hong Kong; Tencent LLM Department
arXiv: 2607.22043 (July 24, 2025)

Key points

Two training paradigms. Mainstream multimodal LLMs (GPT-4V, LLaVA, Kimi) use late fusion: train a text LLM first, then attach a vision encoder (CLIP/SigLIP) through a projection layer. Native multimodal pre-training instead trains a single Transformer from scratch on mixed text-and-image data with one shared parameter set and objective.
Three independent scaling laws. Via IsoFLOP experiments with a Mixture-of-Experts architecture, the authors show that the language objective's compute allocation is nearly insensitive to data composition, while the multimodal objective's optimal allocation depends strongly on the multimodal data ratio.
Pareto frontier as a blueprint. Because both objectives share parameters, fixed compute budgets force trade-offs. The paper characterizes a Pareto frontier of non-dominated configurations (model size, token count, multimodal data proportion) — the first engineering blueprint for native multimodal pre-training.
Positive transfer to text-only spatial reasoning. Natively multimodal models outperform text-only-trained models on text-only spatial reasoning tasks, contradicting the intuition that multimodal training steals capacity from language learning.

Experimental setup

Data: 250B text tokens (web, books, papers) + 75B multimodal tokens (image-text pairs, interleaved documents)
Architecture: MoE decoder-only Transformer with auxiliary-loss-free balancing; images enter via patch embedding as continuous vectors
No vision encoder: no CLIP, no SigLIP — only a patch embedding layer, a fundamental break from late fusion

Engineering implications

1. Language capability is rigid — adding images does not reduce the parameters/tokens language learning requires. 2. Multimodal capability is elastic — the multimodal data ratio directly caps multimodal performance. 3. Native training transfers positively — multimodal training enhances, rather than harms, text spatial reasoning.

Honest assessment

Validation uses only smoothed training loss, since the authors lacked reliable multimodal validation metrics; loss-to-downstream performance mapping is non-linear.
No open-source code has been released.
Conclusions are MoE-specific and may not apply to dense models, where routing mechanics differ.

One-sentence takeaway

Language learning is rigid, multimodal learning is elastic, and in their shared parameter space visual information reinforces language-based spatial reasoning — a "bilingual brain" is multiplication, not addition.

Links:

Paper: https://arxiv.org/abs/2607.22043
HTML version: https://arxiv.org/html/2607.22043v1
Code: not released

FAQ

Who should read this? Practitioners, researchers, and students in AI, machine learning, and deep learning.

What is the core finding? Language and multimodal objectives follow different scaling laws; the paper provides the first Pareto-frontier blueprint for compute allocation in native multimodal pre-training.

Is code available? No, the code has not been open-sourced.