> 📌 This is a GEO-optimized version of the original topic — with a question-driven title, structured data, and FAQ, designed for AI-engine citation.
> One-line takeaway: An analysis of the core findings and engineering implications of "Scaling Native Multimodal Pre-Training From Scratch: Tencent and CUHK found the optimal recipe for a bilingual brain."
Scaling Native Multimodal Pre-Training From Scratch: Tencent and CUHK Found the Optimal Recipe for a "Bilingual Brain"
> Paper: *Scaling Native Multimodal Pre-Training From Scratch* > Authors: Haoyuan Wu, Aoqi Wu, Hai Wang, Jiajia Wu, Jinxiang Ou, Bei Yu > Institutions: The Chinese University of Hong Kong; Tencent LLM Department > arXiv: 2607.22043 (July 24, 2025)
A Counterintuitive Opening
Suppose you want to raise someone fluent in both Chinese and English. Plan A: raise them in a purely Chinese environment until 18, then send them to a US university for four years. Plan B: raise them from birth in a household mixing both languages.
Intuitively, Plan A seems safer — master one language first, then learn the second. But bilingualism research suggests Plan B produces speakers whose two languages are not "translations" of each other but share one conceptual system: thinking in English activates their Chinese semantic network, and vice versa.
Multimodal training of large language models (LLMs) faces exactly the same choice.
The Mainstream Today: Late Fusion's "Translation" Paradigm
Most current multimodal LLMs (GPT-4V, LLaVA, Kimi, etc.) follow Plan A. They train a text-only LLM first, then attach a vision encoder (CLIP, SigLIP), using a projection layer to "translate" visual features into language the LLM understands.
This is efficient but has a fundamental asymmetry: visual and language representations are learned separately, under different data distributions and different objectives. The vision encoder learns via contrastive learning; the LLM learns autoregressive language modeling. They are like two people with different native languages, communicating through a translator (the projection layer).
Native Multimodal: "Bilingual" From Birth
Native multimodal pre-training follows Plan B. From the very beginning, the model trains on mixed text and image data, using one Transformer, one set of parameters, and one optimization objective, so visual and language representations grow within a single representation space.
The idea isn't new — but until now, no one could answer the key engineering question: given a fixed compute budget, how much should go to model parameters, how much to training tokens, and what fraction to multimodal data?
This paper gives the first systematic answer.
Three Independent Scaling Laws
The authors ran IsoFLOP experiments with a Mixture-of-Experts (MoE) architecture: fixing total compute while varying model size and token count to find the optimal configuration per budget.
The key finding: language and multimodal objectives follow different scaling laws.
- The language objective's allocation law is nearly insensitive to data composition. No matter how many images you mix into the training data, the optimal model size and token count for language loss barely change. Language capability is "rigid" — it requires a fixed scale of parameters and tokens and won't shrink because images were added.
- The multimodal objective's allocation law depends strongly on data composition. The higher the multimodal data ratio, the more the optimal configuration for multimodal loss shifts.
- Training data: 250B text tokens (web, books, papers) + 75B multimodal tokens (image-text pairs, interleaved image-text documents)
- Architecture: MoE decoder-only Transformer, auxiliary-loss-free; images projected directly into continuous vectors via patch embedding
- No vision encoder: this is the fundamental difference from late fusion — no CLIP, no SigLIP, just a patch embedding layer
- Validation metric is training loss only. The authors admit that "due to a lack of reliable multimodal validation metrics," they used smoothed training loss as a proxy. The relationship between training loss and downstream performance is not linear, so this proxy may over- or under-estimate real-world effect.
- No open-source code. The paper comes from Tencent's LLM Department, and the code is not public; other teams must build their own MoE training infrastructure to reproduce it.
- MoE conclusions may not apply to dense models. MoE routing may affect how parameters are shared between multimodal and language objectives; dense-model scaling laws may differ.
- A counterintuitive opening
- The mainstream: late fusion's "translation" paradigm
- Native multimodal: "bilingual" from birth
It's like language learning: the grammar and vocabulary needs of Chinese are fixed and don't decrease because you're also learning English; but your English progress depends on how much time you spend in an English environment.
The Pareto Frontier: Compute's "Three-Body Problem"
The harder challenge: language and multimodal objectives share the same parameters. Under a fixed total compute budget, you cannot optimize both simultaneously — a trade-off is required.
The authors characterize this trade-off with a Pareto frontier. Given total compute \(C_{total}\), there exists a set of non-dominated configurations: you cannot improve one goal without hurting the other. This frontier curve is the engineering "recipe table."
The paper provides concrete configuration guidance: for a given compute budget, how large a model to use, how many tokens to train, and what multimodal data proportion. It is the first "construction blueprint" for native multimodal pre-training.
An Unexpected Bonus: Multimodal Training Boosts Text-Only Spatial Reasoning
The most surprising finding comes from downstream evaluation.
The authors found that native multimodal pre-training not only grants multimodal capability but also improves text-only spatial reasoning. A model trained on images + text outperforms a text-only-trained model on spatial reasoning problems presented purely in text.
This challenges the intuition that multimodal training "steals" capacity from language learning. The opposite is true — visual spatial information seems to help the model build stronger spatial concept representations, which can be activated even in text-only tasks.
It's like a person raised bilingually who may be even better at logical reasoning purely in Chinese — because certain conceptual structures from English reinforce their Chinese expression.
Experimental Scale
The experiments are substantial:
Engineering Implications
The paper's engineering value: it turns native multimodal pre-training from "sounds nice but nobody knows how" into "an engineering problem with a blueprint."
For teams training multimodal LLMs (Kimi, Qwen-VL, Emu, SenseNova), these scaling laws offer three directly usable conclusions:
1. Language learning is rigid — don't expect images to save compute on language training. 2. Multimodal learning is elastic — the multimodal data ratio directly determines the ceiling of multimodal capability. 3. Native training transfers positively — multimodal training doesn't harm text capability; it improves spatial reasoning.
An Honest Assessment
Several limitations:
One-Line Summary
The scaling laws for native multimodal pre-training tell us: language learning is rigid, multimodal learning is elastic, and in their shared parameter space, visual information reinforces language-based spatial reasoning in return — a "bilingual brain" is not simple addition, it's multiplication.
---
Paper link: https://arxiv.org/abs/2607.22043 HTML version: https://arxiv.org/html/2607.22043v1 Open-source code: none yet
FAQ
Q1: Who is this for?
Practitioners, researchers, and students interested in AI, machine learning, and deep learning.
Q2: What are the key points?
See the links in the main text.