ComfyUI's Async Weight Streaming: Engineering Magic That Runs Large Models on 4GB VRAM
> Repo: Comfy-Org/ComfyUI > Language: Python > Trending: 155 stars/day
TL;DR
A 4GB VRAM old GPU and a laptop with 8GB RAM running a 12B-parameter diffusion model—not magic, but the result of pushing "data movement" to its limits.
---
The Scenario
You bought an RTX 3070 with 8GB VRAM. Stable Diffusion 1.5 runs fine, SDXL barely works, but FLUX or LTX-2 video models? CUDA out of memory.
The traditional fix is simple—buy a bigger card. RTX 4090 24GB, RTX 5090 32GB. But those cards cost a fortune, and not everyone can afford them.
Over the past few months, the ComfyUI team quietly did something: pushing back the VRAM wall with software, not hardware. They call this "async offloading" combined with "pinned memory", enabled by default in December 2025, free for all NVIDIA GPUs. Sampling speed on an RTX 3070 improved 10-50%, RTX 4070ti improved 20-60%, and RTX 5070ti with NVFP4 quantization doubled outright.
This isn't algorithmic innovation—it's a systems engineering victory.
---
Core Mechanism: Turning "Moving Data" into a Pipeline
Where's the Bottleneck
The bottleneck of diffusion model inference isn't compute—it's bandwidth. A 12B parameter model in fp16 takes 24GB of weights. Your GPU has 8GB of VRAM. What now?
The traditional approach is weight offloading: keep the model in system memory, move each layer to VRAM on demand during inference, then move it out. The problem: transfer speed can't keep up with compute speed—the GPU finishes a layer, then sits idle waiting for the next one.
Analogy: cooking in your kitchen with ingredients in the basement fridge. Each dish takes 30 seconds to cook but 2 minutes to fetch ingredients. The GPU spends most of its time "waiting for ingredients."
Async Offloading: Move Ahead, Don't Wait
ComfyUI's solution is async offload: while the GPU is still computing the current layer, prefetch the next layer's weights from RAM to VRAM.
Back to the kitchen: while you're cooking the first dish, your assistant is already downstairs fetching the second dish's ingredients. When you finish the first, ingredients are already on the counter.
Technically, this requires: 1. Predicting the next layer: knowing the model structure, planning the transfer order ahead of time 2. Asynchronous transfer: using CUDA streams so transfers and compute run in parallel
The key metric is PCIe bandwidth. PCIe 4.0 x16 theoretical bandwidth is 32GB/s, roughly 25GB/s in practice. A 200MB layer takes about 8ms to transfer. If compute time exceeds 8ms, the transfer is completely hidden behind compute—zero overhead.
Pinned Memory: Reducing the Transfer Tax
Moving from system RAM to GPU VRAM passes through the OS. Regular memory is "pageable"—the OS can swap it to disk. GPUs can't directly access pageable memory; it must first be copied to a "pinned" region, then transferred to the GPU. That's an extra copy.
Pinned memory bypasses this copy. ComfyUI pre-allocates a pinned memory staging area, so weights go from regular RAM → pinned memory → VRAM with one fewer hop.
How much does it help? The ComfyUI team's benchmarks show 10-30% sampling speedup on RTX 3070 at 1024x1024, and 20-50% on RTX 4070ti. Crucially—zero configuration, on by default.
DynamicVRAM: Watching the Water Level
The third optimization is DynamicVRAM. Traditional VRAM management is static—you tell the system "reserve 2GB for the model, rest for inference." But different models and resolutions need different amounts, so static allocation often wastes or falls short.
DynamicVRAM does dynamic water-level control: monitoring VRAM pressure in real time, offloading more to RAM when pressure is high, keeping more in VRAM when low. It can even spill weights further to SSD when both VRAM and RAM are full.
This explains the seemingly boastful line in the ComfyUI README: "run even the biggest open source models on as low as 4GB vram + 8GB ram." 4GB VRAM genuinely works—just slower, since most time goes to moving data, not computing.
---
NVFP4: Blackwell's Doubling Boost
If you have an RTX 50-series card, there's another layer of acceleration: NVFP4 quantization.
NVFP4 is a 4-bit floating-point format natively supported by NVIDIA's Blackwell architecture. Compared to fp8 or bf16, weights halve in size and sampling speed doubles. ComfyUI officially supports NVFP4 as of the LTX-2 release.
Benchmark: RTX 5070ti running LTX-2 video, NVFP4 mode is roughly 2x faster than fp8.
But there's a catch—you need PyTorch with CUDA 13.0 (cu130). If your PyTorch is cu124 or older, NVFP4 models will actually be twice as slow as fp8. This is the most common "I used the new feature and got slower" trap.
---
What This Means
For Individual Users
An RTX 3070 can run FLUX and LTX-2—models that previously required 24GB VRAM. Not blazing fast, but it runs. This is a real lowering of the barrier for indie developers, students, and creators without pro GPUs.
For Engineering Practice
ComfyUI's approach highlights an overlooked optimization direction: balancing compute and bandwidth. The bottleneck of large model inference is shifting from "not enough compute" to "can't move data fast enough." GPU compute doubles yearly, but PCIe bandwidth grows slowly. Async offload + pinned memory + dynamic water levels essentially maximize compute utilization in bandwidth-constrained environments.
For the Ecosystem
ComfyUI enables these optimizations by default with no user configuration. That's the right engineering call—most users don't know what pinned memory is, and shouldn't need to. Good infrastructure is invisible.
---
The Numbers
The ComfyUI team benchmarked on RTX 3070 (8GB), RTX 4070ti (12GB), RTX 5070ti (16GB), using Ryzen 5950x + 128GB RAM @ PCIe 4.0 x16:
- RTX 3070: 1024x1024 sampling, 10-30% faster with async offload + pinned memory
- RTX 4070ti: 1328x1328 sampling, 20-50% improvement
- RTX 5070ti: NVFP4 quantization + async offload, roughly 2x speed
---
A Thought
ComfyUI's optimizations point to a broader engineering principle: when hardware upgrades hit a wall, the marginal return on software optimization suddenly rises.
GPU VRAM went from 8GB to 24GB over five years, tripling in price. But async offloading lets 8GB VRAM do what used to require 24GB, at zero cost. This isn't "trading algorithms for hardware" as a compromise—it's "redefining resource boundaries through systems engineering."
Similar ideas apply elsewhere: KV cache offloading to CPU RAM in LLM inference, hot/cold data tiering in databases, page replacement in OSes. The core is the same—when resources are scarce, don't let them idle; overlap movement with computation.
ComfyUI did it, and did it well enough to turn on by default.
---
> Project: https://github.com/Comfy-Org/ComfyUI > Optimization blog: https://blog.comfy.org/p/new-comfyui-optimizations-for-nvidia > Desktop app: Windows, macOS, Linux > License: GPL-3.0