ARM: A 7B Model That Understands, Generates, and Edits Images with One Discrete Representation
Fudan University + ByteDance TikTok + ByteDance Seed propose ARM (AutoRegressive Multimodal Model), breaking the fragmented paradigm of "CLIP for understanding, diffusion for generation" with a unified discrete visual representation.
Background: The "Split Personality" of Multimodal Models
Today's multimodal LLMs split understanding and generation into two worlds:
- Understanding uses CLIP-style continuous representations — strong semantics but weak detail;
- Generation uses diffusion models — pixel-precise but controlled only via text conditioning;
- Editing is messier: either diffusion inpainting or stitched-together external toolchains.
- Initialized from Qwen2.5-7B, plus one linear layer for visual token prediction;
- Standard next-token prediction over unified text + visual token sequences;
- Dynamic resolution via shape tokens inserted into prompts.
- Wang, J., Wang, X., Pan, J., et al., "ARM: An AutoRegressive Large Multimodal Model with Unified Discrete Representations", arXiv:2606.11188, 2026
- Institutions: Fudan University (Trusted Embodied AI), ByteDance TikTok, ByteDance Seed
- Project page: https://github.com/wdrink/ARM
- Base model: ARM-7B (Qwen2.5-7B); tokenizer: SigLIP2-SO400M-512 + FSQ 65K codebook
- Training: ~5T tokens (2.5T PT + 2.5T CT + 0.2B SFT); reward models: GPT-o3 / GPT-4.1
Unified attempts exist (Chameleon, Emu3, Janus), but discrete representations have historically lost to continuous models on understanding and trailed diffusion on generation quality. ARM aims to end that — one 7B autoregressive model, one set of discrete visual tokens, handling understanding, generation, and editing simultaneously.
Core Architecture: Frozen SigLIP2 + Four Losses + 7B AR + GRPO Alignment
Unified Discrete Visual Tokenizer
| Component | Spec | |:---|:---| | Base encoder | SigLIP2-SO400M-512 (frozen) | | Quantization | Finite Scalar Quantization (FSQ), 65K codebook | | Projection | 2 modules, 6 transformer blocks each | | Pixel decoder | 24-block DiT (rectified flow) | | Language model | Frozen 0.5B Qwen2.5 (for caption loss) | | Latent decoder | Initialized from FLUX.1[dev] |
Four supervision objectives (optimized jointly):
| Loss | Purpose | Weight | |:---|:---|:---| | L_cap (Caption) | Language alignment via Qwen2.5 caption cross-entropy | 1 | | L_pix (Pixel reconstruction) | Low-level fidelity via rectified flow | 5 | | L_sig (Sigmoid contrastive) | Match quantized embeddings to SigLIP2 text embeddings | 5 | | L_feat (Feature distillation) | Match quantized features to original SigLIP2 features | 1 |
Total: L_Tok = 1·L_cap + 5·L_pix + 5·L_sig + 1·L_feat
Detokenization uses an FLUX.1-based latent diffusion model (28 steps, CFG 1.5): discrete tokens decode to a latent, then diffuse to pixels.
7B Autoregressive Model
Four-stage training:
| Stage | Tokens | Notes | |:---|:---|:---| | Pretraining (PT) | 2.5T | Native resolution | | Continued training (CT) | 2.5T | Higher res, more interleaved data | | SFT | 0.2B | High-quality instruction data | | RL | – | GRPO alignment for generation and editing |
GRPO Preference Alignment
Reward criteria: for text-to-image — object presence, attributes, spatial relations; for editing — instruction following, preservation of non-target regions, overall visual quality. Reward models: GPT-o3 (generation), GPT-4.1 (editing), KL coefficient 0.01, 16 rollouts.
Results: Wins and Losses Across Three Tasks
Image Understanding
| Benchmark | ARM (7B) | Comparison | |:---|:---|:---| | POPE | 87.3 | Janus-Pro 7B: 87.4; Emu3 8B: 85.2 | | MMBench | 80.7 | LLaVA-OV 7B: 80.8; Qwen2.5-VL 7B: 83.5 | | MMMU | 40.2 | Janus-Pro 7B: 41.0; Qwen2.5-VL 7B: 58.6 | | VQAv2 | 76.1 | BLIP-3o 8B: 83.1 | | SEEDBench | 73.1 | Janus-Pro 7B: 72.1 |
Understanding is comparable to continuous unified models and clearly beats prior discrete unified models (Emu3, Janus). MMMU still lags Qwen2.5-VL (40.2 vs 58.6), showing pure autoregressive reasoning gaps remain.
Image Generation
On GenEval, ARM-RL reaches 86.00 overall DPG vs Qwen-Image (diffusion) at 88.32. RL lifted color accuracy from 0.60→0.90 and position from 0.75→0.89 — RL mainly improves spatial relations and color precision. On WISE (reasoning-based generation), ARM-RL scores 0.56, beating FLUX.1[Dev] (0.50) and BAGEL (0.52).
Image Editing
On GEdit-Bench, ARM achieves perceptual quality (G_PQ) of 7.67–7.68, exceeding all baselines — discrete representations actually have an advantage in editing fidelity (latent-space editing may be more precise than pixel inpainting). After RL, semantic consistency and overall score rise to 6.85/6.68, near Step1X-Edit's 6.70 overall.
Most Surprising Finding: Cross-Task RL Synergy
> Optimizing generation with GRPO also improves editing; optimizing editing also improves generation — and understanding performance is entirely unaffected.
This is cross-task synergy: RL alignment in visual token space is not zero-sum. Semantic features useful for generation also help editing, and fidelity constraints from editing feed back into generation. This hints at a network effect of unified architectures — three tasks sharing one representation space, where optimization on one spills over to the others.
Conclusion: A Feasibility Proof for the Discrete Unified Route
ARM's significance is not SOTA (many metrics aren't first) but proving the unified discrete route is viable.
Key design decisions: 1. Frozen SigLIP2 as a semantic anchor rather than training vision from scratch; 2. Four-way supervision capturing semantics, pixels, language alignment, and feature fidelity; 3. FLUX.1-based detokenizer bridging discrete tokens and pixel space; 4. GRPO alignment directly on visual tokens, yielding cross-task synergy.
Advantages: unified sequence modeling, natural support for interleaved multimodal data (video, webpages), single-model inference. Remaining weaknesses: complex-reasoning gap (MMMU), generation slightly below top diffusion models, and an expensive high-capacity detokenizer.
Discrete representations + a strongly supervised tokenizer + unified autoregressive modeling + RL alignment = one model for understanding, generation, and editing. This is a clear roadmap toward multimodal unification.