Loading…
Glossary term
Splitting a single model across several devices because it is too large for one.
When a model is too large for one device, its parts are placed on different devices. Each device holds and computes only some layers or pieces, and results are passed between them. Pipeline and tensor parallelism are two specific forms.
It is required for models whose weights exceed the memory of a single GPU.