A Better Critic Matters for Distribution Matching Distillation (DMD)

Forum topic · 小凯 · 2026-11-03

Summary

This arXiv paper (2610.42135) investigates what the critic component of Distribution Matching Distillation (DMD) needs—in capacity and in information—to provide a reliable training signal for few-step generative models. On capacity, controlled experiments across seven SiT model sizes on ImageNet show that increasing teacher size while keeping a small critic fixed can degrade student quality and even cause training collapse. The authors propose joint scaling—scaling the critic alongside the teacher—which prevents collapse in their experiments and lets students benefit from larger teachers. On information, they introduce STAC, a step-aware critic conditioned on the student's generation step, estimating step-specific scores instead of forcing the critic to model a step-agnostic mixture of student outputs. Combining joint scaling and STAC consistently improves few-step distillation across class-conditional ImageNet generation, text-to-image generation with SDXL and FLUX.2, and text-to-video generation with Wan2.1. The work offers practical guidance for scaling DMD-based diffusion distillation pipelines.

Overview

Field: Computer Vision Authors: Zongze Wu, Xianbang Wang, Zhengyang Geng, Mingdeng Cao, Zhoutong Zhang, Richard Zhang, Eli Shechtman, Ruiming Cao Published: 2026-10-30 arXiv: 2610.42135

Abstract

Distribution Matching Distillation (DMD) trains a few-step student using the difference between a teacher's score and an online critic's estimate of the student distribution's score. We investigate what the critic needs, in capacity and in information, to provide a reliable training signal.

On the capacity side, controlled experiments across seven SiT model sizes on ImageNet show that increasing teacher size while keeping a small critic fixed can degrade student quality and cause training collapse. Scaling the critic alongside the teacher—a strategy we call joint scaling—prevents collapse in our experiments and allows the student to benefit from larger teachers.

On the information side, rather than forcing the critic to model a step-agnostic mixture of student outputs, we introduce STAC, a step-aware critic conditioned on the student's generation step to estimate step-specific scores.

Joint scaling and STAC together consistently improve few-step distillation on:

Class-conditional ImageNet generation
Text-to-image generation with SDXL and FLUX.2
Text-to-video generation with Wan2.1

Key Takeaways

The critic's capacity is a bottleneck: a small critic can make larger teachers actively harmful, leading to training collapse.
Joint scaling (growing critic size with teacher size) stabilizes training in their experiments.
A step-aware critic (STAC) provides more informative training signals than a step-agnostic critic.
The combination improves few-step distillation across image and video generation benchmarks.

--- *Auto-collected on 2026-11-03*