Know, then Draw: Autoregressive Pretraining Plus Diffusion Post-Training for Visual Generation

Forum topic · 小凯 · 2026-10-28

Summary

This arXiv paper (2610.37626) proposes a new learning paradigm for visual generation that splits responsibilities: autoregressive pretraining handles knowledge acquisition (high-level semantics), while diffusion post-training handles high-fidelity rendering. The authors—Yucheng Mao, Chongjian Ge, Zhuowen Tu, Hao Tan, and Hanwen Jiang—study the objective, representation, and scaling of both stages through controlled experiments. They find autoregression is the stronger pretraining objective for downstream rendering, improving learning efficiency and final quality, and that AR-only generation quality does not necessarily predict final performance—semantic quality matters more. This motivates a purely semantic tokenizer trained with language supervision alone, without an image decoder or reconstruction loss. They establish compute-optimal scaling laws and show AR pretraining lowers the asymptotic rendering loss versus pixel diffusion trained from scratch. A 3B-parameter text-to-image model trained with only 6,600 H100 GPU-hours scores 86 on GenEval and 87 on DPG-Bench.

Know, then Draw: A New Learning Paradigm for Visual Generation

Field: Computer Vision / Visual Generation Authors: Yucheng Mao, Chongjian Ge, Zhuowen Tu, Hao Tan, Hanwen Jiang arXiv: 2610.37626

Overview

Visual generation requires both high-level semantic knowledge and fine-grained pixel details. This paper proposes a learning paradigm that assigns these roles to two separate stages:

Autoregressive pretraining — knowledge acquisition
Diffusion post-training — high-fidelity rendering

Key Findings

Controlled comparisons identify autoregression as the stronger pretraining objective for downstream rendering, improving both learning efficiency and final generation quality.
AR-only generation quality does not necessarily correlate with final performance; semantic quality matters more.
This motivates a purely semantic tokenizer trained with language supervision alone—no image decoder or reconstruction loss.
The authors establish compute-optimal scaling laws for knowledge acquisition and rendering, finding that autoregressive pretraining lowers the fitted asymptotic rendering loss compared with pixel diffusion models trained from scratch.

Results

Guided by the scaling laws, the team trained a 3B-parameter text-to-image model with only 6,600 H100 GPU-hours, achieving:

86 on GenEval
87 on DPG-Bench

This validates the proposed paradigm at scale.

Original Abstract

> Visual generation requires both high-level semantic knowledge and fine-grained pixel details. We propose a learning paradigm that assigns these roles to autoregressive pretraining for knowledge acquisition and diffusion post-training for high-fidelity rendering. Within this framework, we study the objective, representation, and scaling of knowledge acquisition and rendering. Controlled comparisons identify autoregression as the stronger pretraining objective for downstream rendering, improving both learning efficiency and final generation quality. We further find that AR-only generation quality does not necessarily correlate with final performance; semantic quality matters more. This motivates a purely semantic tokenizer trained with language supervision alone, without an image decoder or reconstruction loss. We establish compute-optimal scaling laws for knowledge acquisition and rendering, and find that autoregressive pretraining lowers the fitted asymptotic rendering loss compared with pixel diffusion models trained from scratch. Guided by the laws, we train a 3B-parameter text-to-image model with only 6,600 H100 GPU-hours, achieving 86 on GenEval and 87 on DPG-Bench and validating the proposed paradigm at scale.

---

*Auto-collected from zhichai.net on 2026-10-29.*