Paper Overview
Field: Machine Learning Authors: Iman Khazrak, Narges Nejad, Mostafa M. Rezaee Published: 2026-09-11 arXiv: 2509.05825
Summary
Selecting LoRA rank for diffusion fine-tuning requires balancing quality and compute cost. The authors present a controlled study on CIFAR-10 using a DDPM U-Net with ranks {2,4,8,16,32}, fixed optimization settings, and a reproducible local-folder pytorch-fid protocol.
They report FID, trainable parameters, runtime, and GPU memory, then validate trends with extended-budget DDPM runs (20 epochs; ranks 4/8/16) and a Tiny DiT backbone (10 epochs; ranks 4/8/16).
Key Findings
- Moderate ranks are the most efficient: rank 4 achieves the best DDPM FID (124.1380)
- Rank 8 is close behind (124.2136)
- Higher ranks provide limited gains despite larger adaptation cost
- These findings support small-to-moderate ranks as practical defaults under fixed training budgets
Original Abstract
> Selecting LoRA rank for diffusion fine-tuning requires balancing quality and compute cost. We present a controlled study on CIFAR-10 using a DDPM U-Net with ranks {2,4,8,16,32}, fixed optimization settings, and a reproducible local-folder pytorch-fid protocol. We report FID, trainable parameters, runtime, and GPU memory, then validate trends with extended-budget DDPM runs (20 epochs; ranks 4/8/16) and a Tiny DiT backbone (10 epochs; ranks 4/8/16). Results show moderate ranks are most efficient: rank 4 achieves the best DDPM FID (124.1380), rank 8 is close (124.2136), and higher ranks provide limited gains despite larger adaptation cost. These findings support small-to-moderate ranks as practical defaults under fixed training budgets.
--- *Auto-collected on 2026-09-12*