Multimodal Flow: Unified Flow Modeling of Language and Vision in a Fully Continuous Generative Model

Forum topic · 小凯 · 2026-10-02

Summary

Multimodal Flow is a fully continuous generative model that unifies language and vision under a single flow-matching framework, avoiding the visual quantization bottleneck of discrete-token models and the modality-dependent objectives of hybrid models. It integrates multimodal continuous representations with a shared chunk-causal flow backbone, organizing text blocks and images into ordered continuous hyperchunks that preserve textual token order and visual spatial structure. Joint attention enables cross-modal interaction while modality-specific feed-forward networks handle each modality. Training predicts multiple target chunks in parallel; inference generates hyperchunks sequentially. The authors instantiate MF-1 and pretrain it on multimodal data at 0.6B, 1.2B, and 1.6B scales. Using only 150B pretraining tokens, MF-1 achieves an average score of 82.8 on GenEval and DPG-Bench and 75.3 on VQAv2, MMBench, and POPE, remaining competitive with unified models trained on far more data. Under matched data, optimizer, and parameter budgets, Multimodal Flow outperforms representative hybrid and discrete models.

Paper Overview

Research Area: Computer Vision (CV) Authors: Hongyuan Tao, Xinggang Wang, Lianghui Zhu, et al. Published: 2026-09-30 arXiv: 2609.26725

Abstract

We present Multimodal Flow, a fully continuous generative model of language and vision. Most unified multimodal models either model both language and quantized images as discrete tokens or combine discrete language prediction with continuous image generation. The former introduces a visual quantization bottleneck. The latter requires modality-dependent objectives and sampling procedures. Fully continuous modeling avoids these trade-offs and enables a shared generative process, but remains underexplored for multimodal pretraining.

Multimodal Flow introduces a unified continuous architecture that integrates multimodal continuous representations with a shared chunk-causal flow backbone. It organizes text blocks and images as ordered continuous hyperchunks, preserving textual token order and visual spatial structure. The backbone learns a single vector field over these hyperchunks via Flow Matching. Joint attention enables cross-modal interaction, while modality-specific feed-forward networks process each modality. During training the model predicts multiple target chunks in parallel; at inference it generates hyperchunks sequentially.

Results

The authors instantiate the approach as MF-1 and pretrain it on multimodal data at three scales: 0.6B, 1.2B, and 1.6B parameters. Continued pretraining consistently improves multimodal modeling capability across scales.

With only 150B pretraining tokens, MF-1 achieves an average score of 82.8 on GenEval and DPG-Bench (image generation), and 75.3 on VQAv2, MMBench, and POPE (multimodal understanding)
It remains competitive with unified models trained on substantially more data
Under matched data, optimizer, and parameter budgets, Multimodal Flow further outperforms representative hybrid and discrete models

Key Takeaways

Fully continuous modeling eliminates the visual quantization bottleneck and the need for modality-specific objectives and sampling
Chunk-causal flow matching with hyperchunk organization unifies text and image generation in a single vector field
Strong efficiency: competitive or better performance with only 150B pretraining tokens

中文摘要

我们提出 Multimodal Flow，一个语言与视觉的全连续生成模型。现有统一多模态模型要么将语言和量化图像建模为离散 token（引入视觉量化瓶颈），要么将离散语言预测与连续图像生成结合（需要模态相关目标和采样过程）。全连续建模避免了这些权衡。Multimodal Flow 引入统一连续架构，将多模态连续表示与共享的块因果流主干集成，将文本块和图像组织为有序连续超块（hyperchunk），通过 Flow Matching 学习单一向量场。MF-1 在 0.6B、1.2B、1.6B 三个规模上预训练，仅用 150B token 即在 GenEval/DPG-Bench 平均得分 82.8，在 VQAv2/MMBench/POPE 得分 75.3，与更多数据训练的统一模型保持竞争力。

---

*Auto-collected on 2026-10-02*