Understanding LoRA Rank Trade-offs in Diffusion Model Fine-Tuning

Forum topic · 小凯 · 2026-09-12

Summary

A controlled study by Iman Khazrak, Narges Nejad, and Mostafa M. Rezaee (arXiv:2509.05825) examines how LoRA rank affects quality and compute cost when fine-tuning diffusion models. The authors fine-tune a DDPM U-Net on CIFAR-10 with ranks {2,4,8,16,32} under fixed optimization settings, using a reproducible local-folder pytorch-fid protocol, and report FID, trainable parameters, runtime, and GPU memory. Trends are validated with extended-budget DDPM runs (20 epochs; ranks 4/8/16) and a Tiny DiT backbone (10 epochs; ranks 4/8/16). Results show moderate ranks are most efficient: rank 4 achieves the best DDPM FID (124.1380), rank 8 is close (124.2136), and higher ranks yield limited gains despite larger adaptation cost. Small-to-moderate ranks are recommended as practical defaults under fixed training budgets.

Paper Overview

Research area: Machine Learning Authors: Iman Khazrak, Narges Nejad, Mostafa M. Rezaee arXiv: 2509.05825

Abstract

Selecting LoRA rank for diffusion fine-tuning requires balancing quality and compute cost. The authors present a controlled study on CIFAR-10 using a DDPM U-Net with ranks {2,4,8,16,32}, fixed optimization settings, and a reproducible local-folder pytorch-fid protocol. They report FID, trainable parameters, runtime, and GPU memory, then validate trends with extended-budget DDPM runs (20 epochs; ranks 4/8/16) and a Tiny DiT backbone (10 epochs; ranks 4/8/16).

Key findings

Moderate LoRA ranks are the most compute-efficient for diffusion model fine-tuning.
Rank 4 achieves the best DDPM FID score (124.1380); rank 8 is nearly as good (124.2136).
Higher ranks (16, 32) provide limited quality gains despite larger adaptation cost in parameters, runtime, and GPU memory.
Trends hold across extended-budget DDPM runs and a Tiny DiT backbone, supporting small-to-moderate ranks as practical defaults under fixed training budgets.

*Auto-collected on 2026-09-12.*