ARM: A 7B Autoregressive Model That Understands, Generates, and Edits Images with Unified Discrete Tokens

Forum topic · 小凯 · 2026-06-12

Summary

ARM (AutoRegressive Multimodal Model), developed by Fudan University, ByteDance TikTok, and ByteDance Seed, is a 7B-parameter autoregressive model that unifies image understanding, generation, and editing within a single discrete-token framework. Its core innovation is a unified discrete visual tokenizer built on a frozen SigLIP2-SO400M encoder with 65K-codebook FSQ quantization, trained under four complementary supervision objectives (caption alignment, pixel reconstruction, sigmoid contrastive loss, and feature distillation), paired with a FLUX.1-based latent detokenizer. Initialized from Qwen2.5-7B and trained on ~5T tokens across pretraining, continued training, SFT, and GRPO-based reinforcement learning stages, ARM approaches continuous-representation models on understanding benchmarks (POPE 87.3, MMBench 80.7), nears diffusion models on generation (GenEval 86.00 after RL vs Qwen-Image 88.32), and achieves strong editing quality (GEdit-Bench perceptual quality 7.68). A surprising finding is cross-task synergy: GRPO alignment on generation improves editing and vice versa, without degrading understanding. ARM demonstrates that a fully discrete unified multimodal architecture is viable, though gaps remain in complex reasoning (MMMU 40.2 vs Qwen2.5-VL's 58.6).

ARM (AutoRegressive Multimodal Model) — from Fudan University's Institute of Trustworthy Embodied AI, ByteDance TikTok, and ByteDance Seed — breaks the conventional split where multimodal models use CLIP-style continuous representations for understanding and diffusion models for generation. It is a single 7B autoregressive model with one unified discrete visual representation that simultaneously handles understanding, generation, and editing.

Core Architecture

Unified Discrete Visual Tokenizer

| Component | Spec | |:---|:---| | Base encoder | SigLIP2-SO400M-512 (frozen) | | Quantization | FSQ, 65K codebook | | Projection | 2 groups × 6 transformer blocks | | Pixel decoder | 24-block DiT (rectified flow) | | Language model | Frozen 0.5B Qwen2.5 (caption loss) | | Latent decoder | Initialized from FLUX.1[dev] |

Four complementary supervision objectives are optimized jointly:

L_cap (weight 1): language alignment via Qwen2.5 caption cross-entropy
L_pix (weight 5): pixel-level fidelity via rectified flow
L_sig (weight 5): sigmoid contrastive matching between quantized embeddings and SigLIP2 text embeddings
L_feat (weight 1): feature distillation from raw SigLIP2 visual features

Total: L_Tok = 1·L_cap + 5·L_pix + 5·L_sig + 1·L_feat

Detokenization uses a FLUX.1-based latent diffusion decoder (28 steps, CFG=1.5).

7B Autoregressive Model

Initialized from Qwen2.5-7B, with a linear layer added for visual token prediction
Standard next-token prediction over unified text + visual token sequences
Dynamic resolution via shape tokens inserted into text prompts

Four training stages:

1. Pretraining (PT): 2.5T tokens, native resolution 2. Continued training (CT): 2.5T tokens, higher resolution, more interleaved data 3. SFT: 0.2B tokens, high-quality instruction data 4. RL: GRPO alignment for generation and editing

GRPO Preference Alignment

Text-to-image RL: lr 3×10⁻⁵, 280 steps, GPT-o3 reward (object presence, attributes, spatial relations)
Editing RL: lr 5×10⁻⁵, 100 steps, GPT-4.1 reward (instruction following, non-target region preservation, visual quality)
KL coefficient 0.01, 16 rollouts per step

Results

Understanding

| Benchmark | ARM (7B) | Reference | |:---|:---|:---| | POPE | 87.3 | Janus-Pro 7B: 87.4 | | MMBench | 80.7 | LLaVA-OV 7B: 80.8 | | SEEDBench | 73.1 | Janus-Pro 7B: 72.1 | | MMMU | 40.2 | Qwen2.5-VL 7B: 58.6 |

ARM matches or approaches continuous unified models and clearly outperforms earlier discrete unified models (Emu3, Janus), though complex reasoning (MMMU) still lags.

Generation

GenEval: ARM-RL overall DPG 86.00, close to diffusion-based Qwen-Image (88.32). RL lifted color accuracy from 0.60 → 0.90 and position score from 0.75 → 0.89.
WISE (reasoning-based generation): ARM-RL overall 0.56, exceeding FLUX.1[Dev] (0.50) and BAGEL (0.52).

Editing

On GEdit-Bench, ARM achieves perceptual quality (G_PQ) of 7.67–7.68, above all baselines; after RL, overall score reaches 6.68, near Step1X-Edit's 6.70. The authors attribute the fidelity advantage to latent-space editing being more precise than pixel-level inpainting.

Surprising Finding: Cross-Task RL Synergy

The paper reports (using the word "surprisingly") that GRPO optimization on generation also improves editing — and vice versa — while understanding performance remains unaffected. This suggests a network effect of unified representations: optimizations in the shared discrete visual token space spill over across tasks rather than trading off.

Takeaways

ARM's significance is not SOTA on every metric but a feasibility proof for the discrete unified route:

1. Frozen SigLIP2 as a semantic anchor 2. Four-way supervised tokenizer covering semantics, pixels, language alignment, and feature fidelity 3. FLUX.1-based detokenizer bridging discrete tokens and pixels 4. GRPO on visual tokens producing cross-task synergy

Remaining limitations: complex reasoning gap (MMMU 40.2), generation quality slightly below top diffusion models, and a high-capacity detokenizer adding inference cost.

References

Wang, J., Wang, X., Pan, J., et al., "ARM: An AutoRegressive Large Multimodal Model with Unified Discrete Representations", arXiv:2606.11188, 2026
Project page: https://github.com/wdrink/ARM