The 32 TB Suitcase: What Actually Bottlenecks Training a 2-Trillion-Parameter Model

Two trillion parameters sounds like an astronomical number, but in engineering terms it is first a very concrete arithmetic problem.

How much GPU memory does one parameter occupy during training? Not two bytes — sixteen. Breaking it down: BF16 weights take two bytes, BF16 gradients two bytes, an FP32 master copy of the weights four bytes, Adam optimizer momentum four bytes, and variance another four bytes. Added together, every parameter carries a sixteen-byte suitcase. Two trillion parameters times that: 32 TB.

One H100 has 80 GB of memory. Just fitting the luggage would take four hundred cards — and that's before activations, communication buffers, or any actual computation. So the first lesson of training large models isn't algorithms; it's moving house: how to split up this 32 TB, distribute it, and reassemble it when needed.

First hurdle: memory

ZeRO is the first knife. Its full name is Zero Redundancy Optimizer, and the idea is straightforward: those sixteen bytes of luggage don't need to be carried in full by every card. Optimizer states make up half the total, so slice them across cards, each card storing only its share; gradients can be sliced the same way; the most aggressive version even slices the parameters themselves, gathering them temporarily when needed. Stacking all three schemes can push memory down by more than an order of magnitude — the cost is more communication, and suddenly invisible bandwidth becomes the main character.

Gradient checkpointing is the second knife. During forward computation, neural networks leave behind large intermediate results for backpropagation. With deep layers, these activations can take up more room than the weights themselves. Checkpointing stores only a summary between layers, recomputing each layer during the backward pass. Trading compute for memory slows training by roughly thirty percent, but in exchange activation memory goes from growing linearly with depth to nearly flat.

FlashAttention is the third knife, and the deepest cut. Attention has a square law: double the sequence length and intermediate results quadruple. FlashAttention's approach comes from a hardware observation — GPUs have two tiers of memory: main memory (HBM), large but slow, and on-chip cache (SRAM), small but fast. On an A100, SRAM is only 20 MB, yet its bandwidth is more than ten times that of main memory. So the sequence is split into small blocks, computed inside SRAM, then moved out; the softmax denominator is progressively rescaled across blocks, never touching that huge quadratic matrix. Activation memory grows linearly with sequence length from then on — this cut was a prerequisite for training with million-token contexts later on.

Second hurdle: compute

The classic operation in multi-card training is ring all-reduce: each card computes its gradients, exchanges them pairwise along a ring topology, and after two circuits every card holds the average of all gradients. Mathematically clean, but it only supports fully synchronized training.

There's a subtle phenomenon after gradient averaging: beyond some critical batch size, increasing it further makes the loss-reduction efficiency drop rather than rise — this is the critical batch size. To train with super-critical large batches, plain Adam becomes unstable; the LAMB optimizer sets per-layer step sizes based on each layer's own reliability, pushing the critical point up by an order of magnitude. Half of pre-GPT-3 engineering was wrestling with this critical point.

Then comes precision. BF16 trades FP32's twenty-two mantissa bits for seven, doubling multiplication speed while keeping the dynamic range intact; FP8 goes a step further and was pushed into the lead role in training DeepSeek-V3 — every matrix multiplication in that scheme revolves around fine-grained scaling factors, otherwise the numbers collapse. Every drop in precision gets more compute out of the same cards, provided you're willing to do engineering for numerical stability.

Third hurdle: networking

Four hundred cards can't fit one model, so you have to discuss how to connect them. Within a node, NVLink gives each H100 card 900 GB/s; tensor parallelism splits a large matrix multiplication into small pieces for neighbors, sequence parallelism splits long sequences among neighbors — both require high-frequency communication with neighbors, so they must live in the same machine. Between nodes, bandwidth drops by an order of magnitude, so pipeline parallelism is used: slice the model by layers, transferring activations only once between segments. The 1F1B schedule overlaps one forward pass with one backward pass, filling most of the pipeline bubbles; interleaved pipelines slice segments even finer and feed them in rotation, filling more.

The fork between two routes shows most clearly on MoE. Llama 3 405B takes the dense route: 16,384 H100s with four-way parallelism fully engaged, every card fed to the brim. DeepSeek-V3 is a sparse model with 671B total parameters and 37B activated, using only 2,048 H800s — fewer cards, weaker per card, and inter-card bandwidth cut as well. Its solution nearly abandons tensor parallelism, shifting the communication load to expert parallelism plus DualPipe pipelining, making the assumption that "inter-card communication is cheap" hold from start to finish. The same trillion-scale problem, two answers: one assumes bandwidth is plentiful, the other treats bandwidth as a luxury. Different hardware assumptions, completely different architectures.

Then comes the bill and the failures

Training cost can be roughly estimated with Chinchilla's rule. At a ratio of twenty tokens per parameter, two trillion parameters imply forty trillion tokens; training compute per token is roughly six times the parameter count, for a total of about 4.8×10²⁶ FLOPs. With an H100 at forty percent effective utilization, the full run costs roughly three hundred million GPU-hours. At a cloud rental price of two dollars per hour, that's just over six hundred million dollars; adding storage, networking, redundancy, and electricity, seven hundred fifty million dollars is a reasonable reading of the bill. That number buys not a model itself, but a model plus an entire supply chain of engineering.

What does a real training log look like? In Llama 3 405B's 54-day pretraining, the 16,384 cards suffered 419 unexpected interruptions — on average once every three hours. Of these, 58.7% were GPU-related (including HBM3 memory failures), while the CPU failed only twice over the entire 54 days. The run survived on two things: every H100 had a hot spare card that swapped in immediately, no waiting for a human to move hardware; and automatic checkpoint rollback meant recovery didn't depend on operations staff getting up at midnight. Sixteen thousand cards form a statistical system — a bad card is not an incident, it's the daily routine. Fault tolerance is not a module of the training system; it is the training system.

Looking ahead

Two directions converge at the end. Going up, GB200 NVL72 redefines "within a node" as an entire rack: 72 Blackwell chips linked into one domain via fifth-generation NVLink, 1.8 TB/s per card and 130 TB/s aggregate rack bandwidth — the bandwidth cliff once endured across nodes is partially smoothed out by copper wiring inside a single rack. Going outward, DiLoCo goes the opposite way: synchronizing parameters only every five hundred steps, combined with local optimization. DeepMind's paper measured 500x less communication than fully synchronized training at matched performance — eight disconnected compute islands can be trained as one machine, and an upgraded decoupled version released this April is pushing this route toward transcontinental data centers.

One makes the network fatter, the other makes the conversations rarer. "Hardware-aware co-design," translated, means: first see how expensive the network is, then decide what the architecture looks like. The 32 TB suitcase hasn't gotten smaller — it's just that the people packing it know increasingly well who should carry each layer of luggage, and how often it needs to be moved.