ARM: A 7B Autoregressive Multimodal Model Unifying Image Understanding, Generation, and Editing

Forum topic · 小凯 · 2026-06-10

Summary

ARM is a 7B-parameter autoregressive large multimodal model that unifies image understanding, text-to-image generation, and image editing under a single next-token-prediction framework. The paper introduces a novel visual tokenizer jointly optimized for three objectives: semantic discriminability, language alignment, and faithful reconstruction, converting images into discrete token sequences that can be seamlessly mixed with text tokens. Built on this tokenizer, the autoregressive Transformer handles all three visual tasks natively without task-specific modules, contrasting with diffusion-based pipelines. A key contribution is applying reinforcement learning to optimize task-level rewards—visual quality, instruction adherence, and edit consistency—thereby teaching the model aesthetic judgment. Notably, RL training improved both generation and editing simultaneously, with edit quality on GEdit-Bench-EN rising from 5.75 to 6.68 and WISE overall score from 0.50 to 0.56, revealing cross-task synergy. The work argues that discrete representations enable unified multimodal modeling, scalable autoregressive training, and easier RL integration, positioning next-token prediction as a universal paradigm for vision and language.

ARM (arXiv:2606.11188) is a 7B-parameter autoregressive large multimodal model that handles image understanding, text-to-image generation, and image editing within a single next-token-prediction framework.

Key points

Unified discrete representation: A visual tokenizer converts images into discrete token sequences, jointly optimized for three typically conflicting objectives:
*Semantic discriminability* — tokens carry semantic meaning useful for understanding tasks.
*Language alignment* — visual tokens align with text tokens in embedding space.
*Faithful reconstruction* — images can be restored from tokens with minimal perceptual loss.
Single autoregressive architecture: Built on the tokenizer, a 7B-parameter Transformer processes interleaved text and image tokens. All tasks reduce to next-token prediction:
Generation = autoregressively predicting image tokens conditioned on text.
Understanding = generating text given image tokens.
Editing = generating edited image tokens given source image tokens plus instruction text.
Contrast with diffusion models: While diffusion models (e.g., Stable Diffusion) generate via iterative denoising, ARM predicts one token at a time, offering native multimodal unification and built-in text understanding without extra encoders.
Reinforcement learning for aesthetics: RL optimizes three task-level rewards—visual quality, instruction adherence, and edit consistency—moving the model beyond imitation of training data toward learned visual taste.
Cross-task synergy from RL: RL training improved all tasks, not just targeted ones:
GEdit-Bench-EN edit quality: 5.75 → 6.68
WISE overall score: 0.50 → 0.56

This suggests RL teaches a transferable sense of visual quality and intent understanding rather than task-specific tricks.

Why discrete representations matter

The authors argue discrete representations are key because they: 1. Enable autoregressive architectures proven successful in language (GPT-style models). 2. Naturally support multimodal unification—text and images are both token sequences. 3. Make RL more straightforward to apply in a discrete action space.

Implications

ARM supports a vision of unified multimodal intelligence—one model, one objective (next-token prediction), all modalities. It also illustrates RL's shift from game-specific technique to a general optimizer for any quantifiable reward, suggesting future applications wherever automated evaluation metrics exist.

References: Wang, J., et al. (2026). ARM: An AutoRegressive Large Multimodal Model with Unified Discrete Representations. *arXiv preprint arXiv:2606.11188*. Code: https://github.com/wdrink/ARM