ARM: An AutoRegressive Large Multimodal Model with Unified Discrete Representations

Authors: Junke Wang, Xiao Wang, Jiacheng Pan, Xuefeng Hu, Feng Li, Jingxiang Sun, Chaorui Deng, Zilong Chen, Yunpeng Chen, Kaibin Tian, Matthew Gwilliam, Hao Chen, Danhui Guan, Kun Xu, Weilin Huang, Zuxuan Wu, Haoqi Fan, Yu-Gang Jiang, Zhenheng Yang

arXiv: 2606.11188

Code: https://github.com/wdrink/ARM

Overview

ARM pushes the idea of "visual tokenization" to its logical conclusion: build a top-tier visual codebook that compresses images into short discrete codes, then train a 7B-parameter autoregressive model that can simultaneously understand, generate, and edit images using nothing but next-token prediction. On top of that, the authors use reinforcement learning to teach the model aesthetics — what makes an image look good and whether an edit truly follows user intent.

1. The Visual Tokenizer: An Alphabet for Images

Continuous representations are ill-suited for autregressive models, which predict discrete tokens one at a time. The solution is vector quantization (VQ-VAE-style): an encoder compresses an image into a feature map, each position is approximated by a vector from a finite codebook (e.g., 8,192 entries), and the image becomes a sequence of integer indices — just like text becomes token IDs.

ARM's tokenizer is not a plain VQ-VAE. It jointly optimizes three normally conflicting objectives:

• Semantic discriminability — visual tokens carry semantic meaning (e.g., "cat" maps to a coherent token set), enabling downstream understanding tasks.
• Language alignment — visual tokens align with text tokens in embedding space, forming the basis for captioning and text-to-image generation.
• Faithful reconstruction — the token sequence can be decoded back into high-quality, visually lossless images.

A helpful metaphor: the tokenizer is a translator that must convey meaning accurately, be understood across languages, and preserve every nuance of the original. ARM's tokenizer is one of the few that achieves all three.

2. The 7B Autoregressive Model

With images tokenized, ARM trains a 7B autoregressive Transformer that mixes text and image tokens in one sequence. All tasks collapse into next-token prediction:

• Generation: given text tokens, autoregressively produce image tokens.
• Understanding: given image tokens, produce descriptive text.
• Editing: given original image tokens plus instruction text, produce edited image tokens.

Comparison with Diffusion Models

| Property | Diffusion Models | ARM (autoregressive) | |---|---|---| | Generation | Iterative denoising | Token-by-token prediction | | Inference speed | Slow (many steps) | Relatively fast | | Multimodal unification | Difficult (extra modules) | Natural (all token sequences) | | Text understanding | Usually needs an extra encoder | Natively supported | | Flexibility | High (intervene at any step) | High (intervene at any token) |

ARM demonstrates that with strong discrete representations and proper training, autoregressive models match diffusion models in generation quality while unifying modalities natively.

3. Reinforcement Learning Teaches "Taste"

Maximum-likelihood training teaches a model to imitate training data — but imitating mixed-quality data does not equal producing good images. ARM applies RL to three task-level objectives:

• Visual quality: rewards based on image-quality metrics (e.g., FID, CLIP Score).
• Instruction adherence: text-image matching scores ensure "red cat" prompts do not yield blue cats.
• Edit consistency: edited images must preserve unedited regions while executing instructions.

The most surprising finding is cross-task synergy: RL on one task improves the others. After RL training:

• GEdit-Bench-EN edit quality improved from 5.75 → 6.68
• WISE overall score improved from 0.50 → 0.56

This suggests RL teaches a general "visual taste" and intent-understanding capability that transfers across generation and editing.

4. Architectural Insights

Why discrete representations matter:

1. They enable autoregressive architectures — the paradigm proven at scale by GPT. 2. They natively support multimodal unification — text and images are both token sequences. 3. They make RL straightforward — rewards and policy gradients are more direct in discrete spaces.

And scaling: ARM's results come from a 7B model, but the real ambition is scale — higher-resolution tokenizers, larger parameter counts (7B → 70B), and more training data. Autoregressive models historically improve steadily with scale.

5. Implications

• Unified multimodal intelligence: one model, one objective (next-token prediction), all modalities — a cognitive unification rather than just engineering convenience.
• RL's changing role: RL is evolving from a game-specific technique into a general-purpose optimizer for any task with quantifiable rewards — visual quality, code quality, even aesthetic judgment.

Conclusion

ARM's story: learn a visual alphabet, train a translator that reads and writes it, then give that translator aesthetic training. It marks a step from "AI that can see" toward "AI that understands how to see" and ultimately "AI that creates things worth seeing" — showing that with the right representation, architecture, and optimization objectives, AI can achieve understanding, creation, and judgment of beauty in the visual domain.

References

Wang, J., et al. (2026). ARM: An AutoRegressive Large Multimodal Model with Unified Discrete Representations. *arXiv preprint arXiv:2606.11188*. Code: https://github.com/wdrink/ARM