DeepSpeed Explained: ZeRO Stages, Memory Savings, and Common Misconceptions

Forum topic · 小凯 · 2026-03-27

Summary

This tutorial from the Easy AI series explains DeepSpeed, Microsoft's deep learning optimization library, and its core ZeRO (Zero Redundancy Optimizer) technology. It clarifies a common misconception: GPU memory is not automatically split across multiple cards during distributed training—each GPU stores full model parameters and optimizer states, and multiple GPUs mainly accelerate training speed. The article breaks down ZeRO's three sharding types (parameters, gradients, optimizer states) and compares DeepSpeed Stages 0–3 with concrete per-GPU memory figures: Stage 0 uses about 15.25G, Stage 1 about 12.6G, Stage 2 about 10.4G (recommended), and Stage 3 about 8.15G (maximum savings, with parameters split by layer and fetched on demand). It also covers key features including mixed-precision training, tensor/pipeline/data parallelism, activation recomputation, and memory fragmentation management.

DeepSpeed Explained

1. What is DeepSpeed?

1.1 Core Concept

DeepSpeed is a deep learning optimization library developed by Microsoft. Through ZeRO (Zero Redundancy Optimizer) technology and multiple parallelism strategies, it makes large-scale model training more efficient and easier to use.

1.2 Key Advantages

| Advantage | Description | |-----------|-------------| | GPU memory savings | Per-GPU memory usage drops significantly as GPU count increases | | Training acceleration | Efficient parallelism strategies and memory management | | Model scaling | Supports training of larger models |

1.3 ZeRO Architecture

Model states are sharded across GPUs, eliminating memory redundancy:

| Sharding Type | Description | |---------------|-------------| | Parameter sharding (P) | Model parameters are distributed across GPUs | | Gradient sharding (G) | Distributed computation and storage of gradients | | Optimizer state sharding (O) | Distributed storage and management of optimizer states |

---

2. Common Misconception vs. Reality

2.1 Misconception: Memory Is Split Evenly

❌ Wrong idea:

> "When training on multiple GPUs, memory is automatically divided, so each GPU only carries 1/N of the memory load."

2.2 Reality: Each GPU Is Independent

✅ Actual situation:

> "Every GPU stores the complete model parameters and optimizer states—memory is not automatically split."

> "Multiple GPUs are mainly used to speed up training."

---

3. DeepSpeed Stage Comparison

Stage 0 (Default)

Every GPU stores the full model parameters
Per-GPU memory: ~15.25G

Stage 1

One copy of parameters distributed across all GPUs
Per-GPU memory: ~12.6G

Stage 2 ⭐ Recommended

Parameters shared; optimizer states distributed
Per-GPU memory: ~10.4G
Recommended!

Stage 3

Model split by layers; parameters flow on demand
Per-GPU memory: ~8.15G
Maximum memory savings

---

4. Core Features

ZeRO: sharding of parameters, gradients, and optimizer states
Mixed-precision training: automatic loss scaling and dynamic precision management
Parallelism: combinations of tensor, pipeline, and data parallelism
Memory optimization: activation recomputation and memory fragmentation management

---

Source: Easy AI tutorial series