CARV: Variance Reduction for Expectations with Diffusion Teachers

Forum topic · 小凯 · 2026-05-22

Summary

This post summarizes the arXiv paper 2505.15989 (May 2025) by Jesse Bettencourt, Xindi Wu, and Matan Atzmon in computer vision. Pretrained diffusion models act as frozen teachers whose gradients feed downstream pipelines such as text-to-3D generation, single-step distillation, and data attribution. These teacher gradients are Monte Carlo expectations over noise levels and Gaussian noise samples, and estimator variance dominates compute cost because each draw requires expensive upstream work such as rendering, simulation, and encoding. The authors introduce CARV, a compute-aware variance-accounting framework that motivates a hierarchical Monte Carlo estimator: expensive upstream computation is amortized over cheap diffusion-noise resamples, sharpened by timestep importance sampling and a stratified inverse-CDF construction. In text-to-3D distillation and attribution experiments, CARV achieves 2-3x effective compute multipliers, mostly from amortized reuse, with roughly 25% additional gains from importance sampling plus stratification. In single-step distillation, the same techniques reduce gradient variance by an order of magnitude, yet downstream FID does not improve, indicating a regime where Monte Carlo variance is no longer the bottleneck. The paper is available at https://arxiv.org/abs/2505.15989.

Paper Overview

Field: Computer Vision
Authors: Jesse Bettencourt, Xindi Wu, Matan Atzmon
Published: 2025-05-20
arXiv: 2505.15989

Abstract

Pretrained diffusion models serve as frozen teachers feeding downstream pipelines such as text-to-3D, single-step distillation, and data attribution. The teacher gradients these pipelines consume are Monte Carlo (MC) expectations over noise levels and Gaussian noise samples; their estimator variance dominates compute cost because each draw requires expensive upstream work (rendering, simulation, encoding).

Method: CARV

The authors introduce CARV, a compute-aware variance-accounting framework that motivates a hierarchical MC estimator:

Amortization: reuse the expensive upstream computation over cheap diffusion-noise resamples.
Timestep importance sampling: sharpen estimates across noise levels.
Stratified inverse-CDF construction: further reduce estimator variance.

Results

In text-to-3D distillation and attribution experiments, CARV delivers 2-3x effective compute multipliers, mostly from amortized reuse; roughly 25% additional gains come from importance sampling plus stratification.
In single-step distillation, the same techniques reduce gradient variance by an order of magnitude, but do not improve downstream FID — signaling a regime where MC variance is no longer the bottleneck.