Scaling Laws for Native Multimodal Pre-Training from Scratch: A Tencent and CUHK Study
Forum topic · ✨步子哥 · 2026-07-27
Summary
Researchers from the Chinese University of Hong Kong and Tencent's LLM Department present a systematic study of native multimodal pre-training scaling laws in their paper 'Scaling Native Multimodal Pre-Training From Scratch' (arXiv:2607.22043). Unlike post-hoc fusion methods that attach vision encoders like CLIP or SigLIP to pre-trained text LLMs, native multimodal training learns text and image representations jointly from scratch using a single MoE decoder-only Transformer with patch embeddings and no auxiliary loss. Through IsoFLOP experiments across 250B text tokens and 75B multimodal tokens, the authors discover that language and multimodal objectives follow distinct scaling laws: optimal language-model size and token count remain nearly invariant to data mixture, while multimodal allocation strongly depends on the multimodal data ratio. They derive a Pareto frontier for balancing the two objectives under a fixed compute budget and report a striking transfer effect—native multimodal training improves pure-text spatial reasoning. Limitations include reliance on smoothed training loss as the only validation signal, no released code, and MoE-specific conclusions that may not generalize to dense models.
Key points
- Native vs. post-hoc multimodal training. Current multimodal LLMs such as GPT-4V, LLaVA, and Kimi follow a two-stage pipeline: train a text-only LLM, then attach a frozen vision encoder (CLIP, SigLIP) via a projection layer that "translates" visual features into the LLM's token space. The two towers are optimized with different objectives (contrastive learning vs. autoregressive language modeling), creating an inherent representational asymmetry. Native multimodal pre-training instead trains a single MoE decoder-only Transformer on mixed text and image data from scratch, using patch embeddings—no CLIP, no SigLIP.
- Three independent scaling laws from IsoFLOP experiments. Holding total compute constant and varying model size and token count, the authors find:
- Language loss follows a data-insensitive allocation law: the optimal parameter count and token count barely change as the multimodal data ratio varies. Language capacity is "rigid."
- Multimodal loss follows a data-sensitive allocation law: the optimal configuration shifts substantially with the multimodal data ratio. Multimodal capacity is "elastic."
- Pareto frontier for joint objectives. Because language and multimodal targets share the same parameters, no single configuration optimizes both. The authors map the Pareto frontier and produce a concrete "recipe table" for allocating model size, training tokens, and multimodal data ratio under a given compute budget—the first such blueprint for native multimodal pre-training.
- Surprising transfer: multimodal training boosts pure-text spatial reasoning. Models trained on image+text data outperform text-only baselines on text-only spatial reasoning benchmarks. Visual grounding appears to strengthen spatial concept representations that remain accessible without images, analogous to how bilingual upbringing can reinforce monolateral logical reasoning.
- Experimental scale. 250B text tokens (web, books, papers) plus 75B multimodal tokens (image-text pairs and interleaved image-text documents), trained with an auxiliary-loss-free MoE decoder-only Transformer.
- Engineering implications for teams building Kimi, Qwen-VL, Emu, SenseNova, etc.:
1. Language learning capacity is rigid—do not expect multimodal data to reduce text-training compute.
2. Multimodal learning capacity is elastic—the multimodal data ratio directly sets the ceiling for multimodal ability.
3. Native training yields positive transfer to text, especially spatial reasoning.
- Honest limitations. Validation relies on smoothed training loss only (the authors cite the lack of reliable multimodal validation metrics). No code is released. MoE routing may change how language and multimodal objectives interact, so the scaling laws may not transfer to dense architectures.
Source
- Paper: arXiv:2607.22043 (July 24, 2025)
- HTML: https://arxiv.org/html/2607.22043v1
- Authors: Haoyuan Wu, Aoqi Wu, Hai Wang, Jiajia Wu, Jinxiang Ou, Bei Yu (Chinese University of Hong Kong; Tencent LLM Department)
- Code: not released
This page is an English static mirror generated for search and AI citation.
It may be a full translation or structured summary of the Chinese original.
Canonical interactive discussion lives on the Chinese page:
https://zhichai.net/topic/178503725