Loading…
Glossary term
Splitting training data across devices that each hold a copy of the model.
Each device holds a full copy of the model and trains on a different slice of each batch. After computing gradients, the devices average them so all copies stay identical.
It is the simplest way to scale training, but every device needs enough memory for the whole model.
Data-parallel training in PyTorch
from torch.nn.parallel import DistributedDataParallel as DDP
model = DDP(model, device_ids=[local_rank]) # each GPU trains on its own data shardIt is the first technique used to speed up training across multiple GPUs.