CARV: Variance Reduction for Expectations with Diffusion Teachers

Forum topic · 小凯 · 2026-05-22

Summary

A paper by Jesse Bettencourt, Xindi Wu, and Matan Atzmon (arXiv 2505.15989, May 2025) introduces CARV, a compute-aware variance-accounting framework for pipelines that use pretrained diffusion models as frozen teachers, such as text-to-3D, single-step distillation, and data attribution. These pipelines consume teacher gradients that are Monte Carlo expectations over noise levels and Gaussian noise samples; estimator variance dominates compute cost because each draw requires expensive upstream work like rendering, simulation, or encoding. CARV motivates a hierarchical Monte Carlo estimator that amortizes expensive upstream computation over cheap diffusion-noise resamples, sharpened by timestep importance sampling and a stratified inverse-CDF construction. In text-to-3D distillation and attribution experiments, CARV achieves 2-3x effective compute multipliers without changing the optimization target, with most gains from amortized reuse and roughly 25% additional benefit from importance sampling plus stratification. In single-step distillation, the same techniques reduce gradient variance by an order of magnitude but do not improve downstream FID, indicating a regime where Monte Carlo variance is no longer the bottleneck.

Paper Overview

Field: Computer Vision
Authors: Jesse Bettencourt, Xindi Wu, Matan Atzmon
Published: 2025-05-20
arXiv: 2505.15989

Abstract

Pretrained diffusion models serve as frozen teachers feeding downstream pipelines such as text-to-3D, single-step distillation, and data attribution. The teacher gradients these pipelines consume are Monte Carlo (MC) expectations over noise levels and Gaussian noise samples; their estimator variance dominates compute cost because each draw requires expensive upstream work (rendering, simulation, encoding). The authors introduce CARV, a compute-aware variance-accounting framework that motivates a hierarchical MC estimator: amortize the expensive upstream computation over cheap diffusion-noise resamples, sharpened by timestep importance sampling and a stratified-inverse-CDF construction.

Key Results

In text-to-3D distillation and attribution experiments, CARV delivers 2-3x effective compute multipliers without changing the underlying objective.
Most of the gain comes from amortized reuse of upstream computation; roughly 25% additional comes from importance sampling plus stratification.
In single-step distillation, the same techniques reduce gradient variance by an order of magnitude, but do not improve downstream FID — indicating a regime where MC variance is no longer the bottleneck.

Source

arXiv: https://arxiv.org/abs/2505.15989

*Auto-collected on 2026-05-22*