distributed training

Distributed training splits one model's training run across many accelerators because, past a certain scale, a single GPU or TPU simply cannot hold the job at all — a 175-billion-parameter model like GPT-3 needs roughly 700 gigabytes just to store its weights in 32-bit precision, before counting gradients or optimizer state, which is nearly nine times more than a single H100's 80 gigabytes of memory. Distribution isn't an optimization at this scale, it's the only way the training run fits anywhere, and the specific way a team splits the work — across data, across the model itself, or across both — determines what the resulting cluster can and can't train.

Every GPU holds a full model copy
data parallelism — the simplest, most common form
Each GPU trains on a different batch
same model, different data, in parallel
AllReduce averages gradients across all GPUs
NCCL handles this communication on NVIDIA hardware
Every GPU applies the identical averaged update
all copies stay in sync, step after step

Data parallelism's one hard requirement is that the full model has to fit on a single GPU in the first place, and once a model's weights alone exceed that, adding more GPUs stops helping. Each GPU in a data-parallel setup holds a complete, redundant copy of the entire model, so the approach scales how much data gets processed per step but does nothing to solve the problem of a model too large to fit on one device — a 175-billion-parameter model simply cannot run under plain data parallelism on hardware with 80 gigabytes per GPU, no matter how many of those GPUs are added to the cluster, which is exactly the wall that forced the field toward splitting the model itself.

Tensor and pipeline parallelism split the model across devices in two different directions, and Megatron-LM, NVIDIA's 2019 framework, popularized both inside the same system. Tensor parallelism splits an individual layer's matrix multiplication across multiple GPUs — each GPU computes a slice of the same matrix multiply and the results get combined — which keeps every GPU working on the same layer simultaneously; pipeline parallelism instead assigns different whole layers to different GPUs and streams batches through them like a factory assembly line, so GPU 1 finishes layer 1 of batch A while GPU 2 is already working on layer 2 of batch B. Both exist because splitting a model that's simply too large for one device is a structurally different problem from splitting the data it's trained on.

Redundant Copies vs. Sharded State same 8-GPU cluster, same model — plain data parallel vs. ZeRO-sharded Plain data parallel (DDP) Every GPU stores a COMPLETE, redundant copy of weights, gradients, and optimizer state Model must fit on ONE GPU — more GPUs don't raise that ceiling ZeRO-3 / FSDP (sharded) Each GPU holds only ITS SHARD of weights, gradients, and optimizer state — not the whole thing Rajbhandari et al., 2020 (ZeRO): up to ~8x less memory per GPU Sharding state instead of duplicating it is what lets a cluster train models no single GPU could ever hold.

ZeRO's actual innovation, introduced in Rajbhandari and colleagues' 2020 Microsoft paper "ZeRO: Memory Optimizations Toward Training Trillion Parameter Models," was recognizing that data parallelism's redundant full copies were wasting most of each GPU's memory on duplicate information. Instead of every GPU storing a complete copy of the model's parameters, gradients, and optimizer state, ZeRO shards each of those three components across the GPUs in the cluster, reconstructing whatever a given GPU needs only when it's actually needed for that step's computation — the paper reports its most aggressive sharding level, ZeRO-3, cutting per-GPU memory consumption by roughly a factor of eight compared to standard data parallelism at the same cluster size. PyTorch's FSDP (Fully Sharded Data Parallel) implements essentially the same sharding idea natively, which is why FSDP and DeepSpeed's ZeRO are usually discussed as two implementations of one underlying technique rather than as competing approaches.

The communication backend moving all this data between GPUs is its own infrastructure layer, and on NVIDIA hardware it's almost always NCCL. Every gradient averaging step, every parameter shard reconstruction, and every pipeline hand-off between GPUs has to physically move data between devices, and NCCL — NVIDIA's Collective Communications Library — implements the specific collective operations like AllReduce that distributed training depends on, optimized for NVIDIA's own GPU interconnects; Gloo and MPI exist as alternatives for CPU-only or non-NVIDIA cluster environments, but NCCL's tight integration with NVIDIA hardware is why it's the default choice whenever the cluster is built on NVIDIA GPUs, which most large training runs are.

Every distributed training technique is answering the same underlying question from a different angle: given that one device can't hold or process the whole job, what's the least wasteful way to split it across many — data parallelism splits the workload while duplicating the model, tensor and pipeline parallelism split the model itself across devices, and ZeRO-style sharding eliminates the redundant copies data parallelism otherwise leaves on the table — and real training clusters typically combine several of these techniques at once rather than picking just one.

Go deeper with CFSGPT

Get AI-powered deep-dives, save terms, and run advanced simulations — free account.

Create Free Account