ARM: An AutoRegressive Multimodal Model Unifying Image Understanding, Generation, and Editing with Discrete Representations

Forum topic · 小凯 · 2026-06-11

Summary

ARM is a discrete representation-based autoregressive large multimodal model that unifies image understanding, generation, and editing within a single next-token prediction framework, as presented in arXiv paper 2606.11188 by Junke Wang and colleagues. The system consists of three components: (1) a discrete semantic visual tokenizer that maps images into compact token sequences, supervised with multiple objectives promoting semantic discriminability, language alignment, and faithful reconstruction so diverse tasks share one latent space; (2) a 7B-parameter autoregressive model trained jointly on large-scale text and image token sequences, developing both vision-language perception and generation capabilities; and (3) reinforcement learning applied to text-to-image generation and instruction-guided editing to improve preference alignment. Notably, RL not only improved target task performance (WISE score rising from 0.50 to 0.56) but also induced cross-task synergy. The work was shared on zhichai.net on 2026-06-11.

Overview

ARM is a discrete representation-based AutoRegressive Model that unifies image understanding, generation, and editing within a next-token prediction framework.

Paper: arXiv:2606.11188
Field: Computer Vision
Posted: 2026-06-09

Key Contributions

ARM is built on three main efforts:

1. Discrete semantic visual tokenizer: Maps images into compact token sequences. The tokenizer is supervised with multiple objectives that jointly promote semantic discriminability, language alignment, and faithful reconstruction, thereby supporting diverse tasks in a shared latent space. 2. 7B autoregressive model: Trained over large-scale text and image token sequences, seamlessly developing vision-language perception and generation capabilities. 3. Reinforcement learning: Applied to further improve preference-aligned behavior for text-to-image generation and instruction-guided editing.

Results

Reinforcement learning not only improves the target tasks — for example, the WISE score increases from 0.50 to 0.56 — but also induces cross-task synergy, benefiting overall multimodal capabilities.

Authors

Junke Wang, Xiao Wang, Jiacheng Pan, Xuefeng Hu, Feng Li, Jingxiang Sun, Chaorui Deng, Zilong Chen, Yunpeng Chen, Kaibin Tian, Matthew Gwilliam, Hao Chen, Danhui Guan, Kun Xu, Weilin Huang, Zuxuan Wu, Haoqi Fan, Yu-Gang Jiang, and Zhenheng Yang.